2024-03-29T11:01:30.833 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-29T11:01:30.834 DEBUG:teuthology.run:Teuthology command: teuthology --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} --name amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi --owner scheduled_amathuri@teuthology --archive /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622679 --verbose -- /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622679/orig.config.yaml 2024-03-29T11:01:30.869 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:01:30.941 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622679 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} email: null first_in_suite: false job_id: '7622679' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: lockdep: true mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: zstd bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 8152 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi047.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPNhVwhJk+rZpOqy4vRT+1shz5VX6cJK93bgh6UFH/YgmVvQNabahQprKdUnKhgDVpyMN5spp1rihixflu+VieQ= smithi121.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMS5EAtFB8SYN2sAGTqa/rrYY+xvMfYBmHUjp0wioVu/18xT7z+V0ACprXjanFGxqrTGH9apissqBCVGhTCTp84= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-26_05:45:44 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-29T11:01:30.941 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-29T11:01:30.942 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-29T11:01:30.942 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-29T11:01:30.944 INFO:teuthology.task.internal:Checking packages... 2024-03-29T11:01:30.967 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-29T11:01:30.967 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-29T11:01:30.967 INFO:teuthology.packaging:ref: None 2024-03-29T11:01:30.967 INFO:teuthology.packaging:tag: None 2024-03-29T11:01:30.968 INFO:teuthology.packaging:branch: main 2024-03-29T11:01:30.968 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:01:30.968 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-29T11:01:31.224 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2530.gb74786fc 2024-03-29T11:01:31.226 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-29T11:01:31.233 INFO:teuthology.task.internal:no buildpackages task found 2024-03-29T11:01:31.233 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-29T11:01:31.998 INFO:teuthology.task.internal:Saving configuration 2024-03-29T11:01:32.011 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-29T11:01:32.018 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:01:32.045 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi047.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622679', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:56:31.412883', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPNhVwhJk+rZpOqy4vRT+1shz5VX6cJK93bgh6UFH/YgmVvQNabahQprKdUnKhgDVpyMN5spp1rihixflu+VieQ='} 2024-03-29T11:01:32.069 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi121.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622679', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:56:31.410886', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMS5EAtFB8SYN2sAGTqa/rrYY+xvMfYBmHUjp0wioVu/18xT7z+V0ACprXjanFGxqrTGH9apissqBCVGhTCTp84='} 2024-03-29T11:01:32.070 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-29T11:01:32.076 INFO:teuthology.task.internal:roles: ubuntu@smithi047.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-29T11:01:32.076 INFO:teuthology.task.internal:roles: ubuntu@smithi121.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-29T11:01:32.076 INFO:teuthology.run_tasks:Running task console_log... 2024-03-29T11:01:32.165 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fbc4600c0d0>, signals=[15]) 2024-03-29T11:01:32.165 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-29T11:01:32.177 INFO:teuthology.task.internal:Opening connections... 2024-03-29T11:01:32.177 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi047.front.sepia.ceph.com 2024-03-29T11:01:32.180 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:01:32.350 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi121.front.sepia.ceph.com 2024-03-29T11:01:32.351 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:01:32.449 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-29T11:01:32.503 DEBUG:teuthology.orchestra.run.smithi047:> uname -m 2024-03-29T11:01:32.523 INFO:teuthology.orchestra.run.smithi047.stdout:x86_64 2024-03-29T11:01:32.523 DEBUG:teuthology.orchestra.run.smithi047:> cat /etc/os-release 2024-03-29T11:01:32.584 INFO:teuthology.orchestra.run.smithi047.stdout:NAME="CentOS Stream" 2024-03-29T11:01:32.585 INFO:teuthology.orchestra.run.smithi047.stdout:VERSION="9" 2024-03-29T11:01:32.585 INFO:teuthology.orchestra.run.smithi047.stdout:ID="centos" 2024-03-29T11:01:32.585 INFO:teuthology.orchestra.run.smithi047.stdout:ID_LIKE="rhel fedora" 2024-03-29T11:01:32.585 INFO:teuthology.orchestra.run.smithi047.stdout:VERSION_ID="9" 2024-03-29T11:01:32.585 INFO:teuthology.orchestra.run.smithi047.stdout:PLATFORM_ID="platform:el9" 2024-03-29T11:01:32.585 INFO:teuthology.orchestra.run.smithi047.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T11:01:32.585 INFO:teuthology.orchestra.run.smithi047.stdout:ANSI_COLOR="0;31" 2024-03-29T11:01:32.585 INFO:teuthology.orchestra.run.smithi047.stdout:LOGO="fedora-logo-icon" 2024-03-29T11:01:32.585 INFO:teuthology.orchestra.run.smithi047.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T11:01:32.585 INFO:teuthology.orchestra.run.smithi047.stdout:HOME_URL="https://centos.org/" 2024-03-29T11:01:32.585 INFO:teuthology.orchestra.run.smithi047.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T11:01:32.585 INFO:teuthology.orchestra.run.smithi047.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T11:01:32.586 INFO:teuthology.orchestra.run.smithi047.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T11:01:32.586 INFO:teuthology.lock.ops:Updating smithi047.front.sepia.ceph.com on lock server 2024-03-29T11:01:32.610 DEBUG:teuthology.orchestra.run.smithi121:> uname -m 2024-03-29T11:01:32.629 INFO:teuthology.orchestra.run.smithi121.stdout:x86_64 2024-03-29T11:01:32.629 DEBUG:teuthology.orchestra.run.smithi121:> cat /etc/os-release 2024-03-29T11:01:32.689 INFO:teuthology.orchestra.run.smithi121.stdout:NAME="CentOS Stream" 2024-03-29T11:01:32.690 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION="9" 2024-03-29T11:01:32.690 INFO:teuthology.orchestra.run.smithi121.stdout:ID="centos" 2024-03-29T11:01:32.690 INFO:teuthology.orchestra.run.smithi121.stdout:ID_LIKE="rhel fedora" 2024-03-29T11:01:32.690 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION_ID="9" 2024-03-29T11:01:32.690 INFO:teuthology.orchestra.run.smithi121.stdout:PLATFORM_ID="platform:el9" 2024-03-29T11:01:32.690 INFO:teuthology.orchestra.run.smithi121.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T11:01:32.690 INFO:teuthology.orchestra.run.smithi121.stdout:ANSI_COLOR="0;31" 2024-03-29T11:01:32.690 INFO:teuthology.orchestra.run.smithi121.stdout:LOGO="fedora-logo-icon" 2024-03-29T11:01:32.690 INFO:teuthology.orchestra.run.smithi121.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T11:01:32.690 INFO:teuthology.orchestra.run.smithi121.stdout:HOME_URL="https://centos.org/" 2024-03-29T11:01:32.690 INFO:teuthology.orchestra.run.smithi121.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T11:01:32.690 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T11:01:32.691 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T11:01:32.691 INFO:teuthology.lock.ops:Updating smithi121.front.sepia.ceph.com on lock server 2024-03-29T11:01:32.719 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-29T11:01:32.732 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-29T11:01:32.738 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-29T11:01:32.739 DEBUG:teuthology.orchestra.run.smithi047:> test '!' -e /home/ubuntu/cephtest 2024-03-29T11:01:32.743 DEBUG:teuthology.orchestra.run.smithi121:> test '!' -e /home/ubuntu/cephtest 2024-03-29T11:01:32.761 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-29T11:01:32.770 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-29T11:01:32.770 DEBUG:teuthology.orchestra.run.smithi047:> test -z $(ls -A /var/lib/ceph) 2024-03-29T11:01:32.800 DEBUG:teuthology.orchestra.run.smithi121:> test -z $(ls -A /var/lib/ceph) 2024-03-29T11:01:32.908 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-29T11:01:32.955 INFO:teuthology.run_tasks:Running task kernel... 2024-03-29T11:01:32.969 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-29T11:01:32.969 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-29T11:01:32.970 DEBUG:teuthology.orchestra.run.smithi047:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T11:01:32.970 DEBUG:teuthology.orchestra.run.smithi121:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T11:01:32.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:01:32.987 DEBUG:teuthology.orchestra.run.smithi121:> uname -r 2024-03-29T11:01:32.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:01:32.988 DEBUG:teuthology.orchestra.run.smithi047:> uname -r 2024-03-29T11:01:33.043 INFO:teuthology.orchestra.run.smithi121.stdout:5.14.0-361.el9.x86_64 2024-03-29T11:01:33.044 INFO:teuthology.task.kernel:Running kernel on smithi121: 5.14.0-361.el9.x86_64 2024-03-29T11:01:33.044 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum install -y kernel 2024-03-29T11:01:33.045 INFO:teuthology.orchestra.run.smithi047.stdout:5.14.0-361.el9.x86_64 2024-03-29T11:01:33.045 INFO:teuthology.task.kernel:Running kernel on smithi047: 5.14.0-361.el9.x86_64 2024-03-29T11:01:33.045 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum install -y kernel 2024-03-29T11:01:35.084 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-29T11:01:35.213 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-29T11:01:36.550 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS-9 - BaseOS 5.9 MB/s | 2.0 MB 00:00 2024-03-29T11:01:36.729 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS-9 - BaseOS 6.0 MB/s | 2.0 MB 00:00 2024-03-29T11:01:37.929 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS Stream 9 - BaseOS 9.5 MB/s | 8.0 MB 00:00 2024-03-29T11:01:38.591 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - BaseOS 4.7 MB/s | 8.0 MB 00:01 2024-03-29T11:01:40.728 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - AppStream 18 MB/s | 19 MB 00:01 2024-03-29T11:01:44.859 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - CRB 9.9 MB/s | 6.1 MB 00:00 2024-03-29T11:01:46.460 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 16 kB 00:00 2024-03-29T11:01:48.469 INFO:teuthology.orchestra.run.smithi121.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-29T11:01:53.477 INFO:teuthology.orchestra.run.smithi121.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-29T11:01:54.045 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS Stream 9 - AppStream 1.3 MB/s | 19 MB 00:14 2024-03-29T11:01:55.808 INFO:teuthology.orchestra.run.smithi121.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T11:01:55.809 INFO:teuthology.orchestra.run.smithi121.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T11:01:55.897 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-29T11:01:55.902 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:01:55.903 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-03-29T11:01:55.903 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:01:55.903 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-29T11:01:55.903 INFO:teuthology.orchestra.run.smithi121.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T11:01:55.903 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-03-29T11:01:55.903 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T11:01:55.903 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T11:01:55.903 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T11:01:55.903 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:01:55.903 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-29T11:01:55.903 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:01:55.903 INFO:teuthology.orchestra.run.smithi121.stdout:Install 4 Packages 2024-03-29T11:01:55.903 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:01:55.904 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 76 M 2024-03-29T11:01:55.904 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 123 M 2024-03-29T11:01:55.904 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-29T11:01:56.061 INFO:teuthology.orchestra.run.smithi121.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-03-29T11:01:56.645 INFO:teuthology.orchestra.run.smithi121.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 20 MB/s | 15 MB 00:00 2024-03-29T11:01:57.554 INFO:teuthology.orchestra.run.smithi121.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 20 MB/s | 34 MB 00:01 2024-03-29T11:01:57.804 INFO:teuthology.orchestra.run.smithi121.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 16 MB/s | 28 MB 00:01 2024-03-29T11:01:57.807 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:01:57.807 INFO:teuthology.orchestra.run.smithi121.stdout:Total 40 MB/s | 76 MB 00:01 2024-03-29T11:01:57.807 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-29T11:01:58.145 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.1 MB 00:00 2024-03-29T11:01:58.167 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-29T11:01:58.167 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-29T11:01:58.999 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-29T11:01:58.999 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-29T11:01:59.972 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS Stream 9 - Extras packages 25 kB/s | 16 kB 00:00 2024-03-29T11:02:00.914 INFO:teuthology.orchestra.run.smithi047.stdout:Extra Packages for Enterprise Linux 26 MB/s | 21 MB 00:00 2024-03-29T11:02:01.295 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-29T11:02:01.885 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:02:01.979 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:02:04.711 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:02:04.848 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:02:06.214 INFO:teuthology.orchestra.run.smithi047.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-03-29T11:02:08.579 INFO:teuthology.orchestra.run.smithi047.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T11:02:08.580 INFO:teuthology.orchestra.run.smithi047.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T11:02:08.681 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-29T11:02:08.689 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-29T11:02:08.689 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2024-03-29T11:02:08.689 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-29T11:02:08.689 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-29T11:02:08.689 INFO:teuthology.orchestra.run.smithi047.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T11:02:08.689 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2024-03-29T11:02:08.689 INFO:teuthology.orchestra.run.smithi047.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T11:02:08.690 INFO:teuthology.orchestra.run.smithi047.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T11:02:08.690 INFO:teuthology.orchestra.run.smithi047.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T11:02:08.690 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:02:08.690 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-29T11:02:08.690 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-29T11:02:08.690 INFO:teuthology.orchestra.run.smithi047.stdout:Install 4 Packages 2024-03-29T11:02:08.690 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:02:08.691 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 76 M 2024-03-29T11:02:08.691 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 123 M 2024-03-29T11:02:08.691 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-29T11:02:08.924 INFO:teuthology.orchestra.run.smithi047.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 823 kB/s | 192 kB 00:00 2024-03-29T11:02:09.574 INFO:teuthology.orchestra.run.smithi047.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 17 MB/s | 15 MB 00:00 2024-03-29T11:02:09.579 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:02:09.629 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:02:10.008 INFO:teuthology.orchestra.run.smithi047.stdout:(3/4): kernel-modules-core-5.14.0-434.el9.x86_6 26 MB/s | 28 MB 00:01 2024-03-29T11:02:10.699 INFO:teuthology.orchestra.run.smithi047.stdout:(4/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 17 MB/s | 34 MB 00:02 2024-03-29T11:02:10.703 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:02:10.703 INFO:teuthology.orchestra.run.smithi047.stdout:Total 38 MB/s | 76 MB 00:02 2024-03-29T11:02:10.703 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-29T11:02:11.162 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-29T11:02:11.162 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-29T11:02:12.038 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-29T11:02:12.039 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-29T11:02:14.469 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-29T11:02:14.498 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:02:15.090 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:02:15.168 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:02:18.072 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:02:18.202 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:02:23.273 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:02:23.310 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:02:28.406 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:02:43.805 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:02:43.805 INFO:teuthology.orchestra.run.smithi121.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T11:02:43.805 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:02:43.848 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:02:44.161 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:02:44.161 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:02:44.161 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:02:44.161 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:02:44.607 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:02:44.607 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:02:44.607 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-29T11:02:44.607 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T11:02:44.607 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T11:02:44.607 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T11:02:44.607 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T11:02:44.607 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:02:44.608 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-29T11:02:44.908 DEBUG:teuthology.orchestra.run.smithi121:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:02:45.031 INFO:teuthology.orchestra.run.smithi121.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:02:45.031 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:02:45.031 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T11:02:45.031 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T11:02:45.031 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T11:02:45.031 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-29T11:02:45.032 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi121.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T11:02:45.032 DEBUG:teuthology.orchestra.run.smithi121:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:02:45.111 INFO:teuthology.orchestra.run.smithi121.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:02:45.112 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:02:45.112 DEBUG:teuthology.orchestra.run.smithi121:> sudo rpm -qi grub2-tools 2024-03-29T11:02:45.146 INFO:teuthology.orchestra.run.smithi121.stdout:Name : grub2-tools 2024-03-29T11:02:45.146 INFO:teuthology.orchestra.run.smithi121.stdout:Epoch : 1 2024-03-29T11:02:45.146 INFO:teuthology.orchestra.run.smithi121.stdout:Version : 2.06 2024-03-29T11:02:45.146 INFO:teuthology.orchestra.run.smithi121.stdout:Release : 68.el9 2024-03-29T11:02:45.146 INFO:teuthology.orchestra.run.smithi121.stdout:Architecture: x86_64 2024-03-29T11:02:45.146 INFO:teuthology.orchestra.run.smithi121.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T11:02:45.146 INFO:teuthology.orchestra.run.smithi121.stdout:Group : Unspecified 2024-03-29T11:02:45.147 INFO:teuthology.orchestra.run.smithi121.stdout:Size : 8264254 2024-03-29T11:02:45.147 INFO:teuthology.orchestra.run.smithi121.stdout:License : GPLv3+ 2024-03-29T11:02:45.147 INFO:teuthology.orchestra.run.smithi121.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T11:02:45.147 INFO:teuthology.orchestra.run.smithi121.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T11:02:45.147 INFO:teuthology.orchestra.run.smithi121.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T11:02:45.147 INFO:teuthology.orchestra.run.smithi121.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T11:02:45.147 INFO:teuthology.orchestra.run.smithi121.stdout:Packager : builder@centos.org 2024-03-29T11:02:45.147 INFO:teuthology.orchestra.run.smithi121.stdout:Vendor : CentOS 2024-03-29T11:02:45.147 INFO:teuthology.orchestra.run.smithi121.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T11:02:45.147 INFO:teuthology.orchestra.run.smithi121.stdout:Summary : Support tools for GRUB. 2024-03-29T11:02:45.147 INFO:teuthology.orchestra.run.smithi121.stdout:Description : 2024-03-29T11:02:45.147 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:02:45.147 INFO:teuthology.orchestra.run.smithi121.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T11:02:45.147 INFO:teuthology.orchestra.run.smithi121.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T11:02:45.148 INFO:teuthology.orchestra.run.smithi121.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T11:02:45.148 INFO:teuthology.orchestra.run.smithi121.stdout:hardware devices. 2024-03-29T11:02:45.148 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:02:45.148 INFO:teuthology.orchestra.run.smithi121.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T11:02:45.149 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T11:02:45.149 INFO:teuthology.task.kernel:Updating grub on smithi121 to boot 5.14.0-434.el9.x86_64 2024-03-29T11:02:45.149 DEBUG:teuthology.orchestra.run.smithi121:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T11:02:45.829 INFO:teuthology.orchestra.run.smithi121.stderr:Generating grub configuration file ... 2024-03-29T11:02:47.203 INFO:teuthology.orchestra.run.smithi121.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T11:02:47.223 INFO:teuthology.orchestra.run.smithi121.stderr:done 2024-03-29T11:02:47.225 DEBUG:teuthology.orchestra.run.smithi121:> mktemp 2024-03-29T11:02:47.240 INFO:teuthology.orchestra.run.smithi121.stdout:/tmp/tmp.LuKhhbkorq 2024-03-29T11:02:47.241 DEBUG:teuthology.orchestra.run.smithi121:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.LuKhhbkorq 2024-03-29T11:02:47.309 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0666 /tmp/tmp.LuKhhbkorq 2024-03-29T11:02:47.441 DEBUG:teuthology.orchestra.remote:smithi121:/tmp/tmp.LuKhhbkorq is 6KB 2024-03-29T11:02:47.490 DEBUG:teuthology.orchestra.run.smithi121:> rm -fr /tmp/tmp.LuKhhbkorq 2024-03-29T11:02:47.504 DEBUG:teuthology.orchestra.run.smithi121:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T11:02:47.574 INFO:teuthology.orchestra.run.smithi121.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T11:02:47.574 INFO:teuthology.orchestra.run.smithi121.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T11:02:47.575 INFO:teuthology.orchestra.run.smithi121.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T11:02:47.575 INFO:teuthology.orchestra.run.smithi121.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T11:02:47.576 DEBUG:teuthology.orchestra.run.smithi121:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T11:02:47.729 DEBUG:teuthology.orchestra.run.smithi121:> sudo shutdown -r now 2024-03-29T11:02:59.207 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:02:59.208 INFO:teuthology.orchestra.run.smithi047.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T11:02:59.208 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:02:59.253 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:02:59.648 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:02:59.649 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:02:59.649 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:02:59.649 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:03:00.055 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:03:00.055 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:03:00.055 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-29T11:03:00.055 INFO:teuthology.orchestra.run.smithi047.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T11:03:00.055 INFO:teuthology.orchestra.run.smithi047.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T11:03:00.056 INFO:teuthology.orchestra.run.smithi047.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T11:03:00.056 INFO:teuthology.orchestra.run.smithi047.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T11:03:00.056 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:03:00.056 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-29T11:03:00.384 DEBUG:teuthology.orchestra.run.smithi047:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:03:00.493 INFO:teuthology.orchestra.run.smithi047.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:03:00.493 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:03:00.493 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T11:03:00.493 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T11:03:00.493 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T11:03:00.494 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-29T11:03:00.494 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi047.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T11:03:00.494 DEBUG:teuthology.orchestra.run.smithi047:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:03:00.577 INFO:teuthology.orchestra.run.smithi047.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:03:00.577 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:03:00.577 DEBUG:teuthology.orchestra.run.smithi047:> sudo rpm -qi grub2-tools 2024-03-29T11:03:00.614 INFO:teuthology.orchestra.run.smithi047.stdout:Name : grub2-tools 2024-03-29T11:03:00.615 INFO:teuthology.orchestra.run.smithi047.stdout:Epoch : 1 2024-03-29T11:03:00.615 INFO:teuthology.orchestra.run.smithi047.stdout:Version : 2.06 2024-03-29T11:03:00.615 INFO:teuthology.orchestra.run.smithi047.stdout:Release : 68.el9 2024-03-29T11:03:00.615 INFO:teuthology.orchestra.run.smithi047.stdout:Architecture: x86_64 2024-03-29T11:03:00.615 INFO:teuthology.orchestra.run.smithi047.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T11:03:00.615 INFO:teuthology.orchestra.run.smithi047.stdout:Group : Unspecified 2024-03-29T11:03:00.615 INFO:teuthology.orchestra.run.smithi047.stdout:Size : 8264254 2024-03-29T11:03:00.615 INFO:teuthology.orchestra.run.smithi047.stdout:License : GPLv3+ 2024-03-29T11:03:00.615 INFO:teuthology.orchestra.run.smithi047.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T11:03:00.615 INFO:teuthology.orchestra.run.smithi047.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T11:03:00.615 INFO:teuthology.orchestra.run.smithi047.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T11:03:00.615 INFO:teuthology.orchestra.run.smithi047.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T11:03:00.615 INFO:teuthology.orchestra.run.smithi047.stdout:Packager : builder@centos.org 2024-03-29T11:03:00.616 INFO:teuthology.orchestra.run.smithi047.stdout:Vendor : CentOS 2024-03-29T11:03:00.616 INFO:teuthology.orchestra.run.smithi047.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T11:03:00.616 INFO:teuthology.orchestra.run.smithi047.stdout:Summary : Support tools for GRUB. 2024-03-29T11:03:00.616 INFO:teuthology.orchestra.run.smithi047.stdout:Description : 2024-03-29T11:03:00.616 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:03:00.616 INFO:teuthology.orchestra.run.smithi047.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T11:03:00.616 INFO:teuthology.orchestra.run.smithi047.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T11:03:00.616 INFO:teuthology.orchestra.run.smithi047.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T11:03:00.616 INFO:teuthology.orchestra.run.smithi047.stdout:hardware devices. 2024-03-29T11:03:00.616 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:03:00.616 INFO:teuthology.orchestra.run.smithi047.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T11:03:00.617 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T11:03:00.617 INFO:teuthology.task.kernel:Updating grub on smithi047 to boot 5.14.0-434.el9.x86_64 2024-03-29T11:03:00.617 DEBUG:teuthology.orchestra.run.smithi047:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T11:03:01.227 INFO:teuthology.orchestra.run.smithi047.stderr:Generating grub configuration file ... 2024-03-29T11:03:02.579 INFO:teuthology.orchestra.run.smithi047.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T11:03:02.599 INFO:teuthology.orchestra.run.smithi047.stderr:done 2024-03-29T11:03:02.602 DEBUG:teuthology.orchestra.run.smithi047:> mktemp 2024-03-29T11:03:02.617 INFO:teuthology.orchestra.run.smithi047.stdout:/tmp/tmp.Ub5PQfzFtY 2024-03-29T11:03:02.617 DEBUG:teuthology.orchestra.run.smithi047:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Ub5PQfzFtY 2024-03-29T11:03:02.689 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 0666 /tmp/tmp.Ub5PQfzFtY 2024-03-29T11:03:02.824 DEBUG:teuthology.orchestra.remote:smithi047:/tmp/tmp.Ub5PQfzFtY is 6KB 2024-03-29T11:03:02.873 DEBUG:teuthology.orchestra.run.smithi047:> rm -fr /tmp/tmp.Ub5PQfzFtY 2024-03-29T11:03:02.887 DEBUG:teuthology.orchestra.run.smithi047:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T11:03:02.959 INFO:teuthology.orchestra.run.smithi047.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T11:03:02.960 INFO:teuthology.orchestra.run.smithi047.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T11:03:02.960 INFO:teuthology.orchestra.run.smithi047.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T11:03:02.960 INFO:teuthology.orchestra.run.smithi047.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T11:03:02.961 DEBUG:teuthology.orchestra.run.smithi047:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T11:03:03.125 DEBUG:teuthology.orchestra.run.smithi047:> sudo shutdown -r now 2024-03-29T11:03:17.746 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-29T11:03:17.747 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-03-29T11:03:17.748 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:03:33.142 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-29T11:03:33.143 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi047.front.sepia.ceph.com' 2024-03-29T11:03:33.143 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:03:36.222 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.121 2024-03-29T11:03:45.233 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-03-29T11:03:45.234 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:03:48.286 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.121 2024-03-29T11:03:51.582 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.47 2024-03-29T11:04:00.294 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-03-29T11:04:00.295 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:04:00.584 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi047.front.sepia.ceph.com' 2024-03-29T11:04:00.584 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:04:32.258 DEBUG:teuthology.orchestra.run.smithi121:> true 2024-03-29T11:04:32.796 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-03-29T11:04:32.796 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:04:32.796 DEBUG:teuthology.orchestra.run.smithi121:> uname -r 2024-03-29T11:04:32.825 INFO:teuthology.orchestra.run.smithi121.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:04:32.825 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:04:32.825 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:04:32.825 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:04:33.827 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-29T11:04:33.827 DEBUG:teuthology.orchestra.run.smithi121:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:04:33.886 INFO:teuthology.orchestra.run.smithi121.stdout:ttyS1 2024-03-29T11:04:34.066 DEBUG:teuthology.parallel:result is None 2024-03-29T11:05:00.587 DEBUG:teuthology.orchestra.remote:timed out 2024-03-29T11:05:12.588 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi047.front.sepia.ceph.com' 2024-03-29T11:05:12.589 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:05:12.889 DEBUG:teuthology.orchestra.run.smithi047:> true 2024-03-29T11:05:13.284 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi047.front.sepia.ceph.com' 2024-03-29T11:05:13.284 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:05:13.284 DEBUG:teuthology.orchestra.run.smithi047:> uname -r 2024-03-29T11:05:13.300 INFO:teuthology.orchestra.run.smithi047.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:05:13.300 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:05:13.300 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:05:13.300 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:05:14.301 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-29T11:05:14.301 DEBUG:teuthology.orchestra.run.smithi047:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:05:14.358 INFO:teuthology.orchestra.run.smithi047.stdout:ttyS1 2024-03-29T11:05:14.392 DEBUG:teuthology.parallel:result is None 2024-03-29T11:05:14.393 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-29T11:05:14.402 INFO:teuthology.task.internal:Creating test directory... 2024-03-29T11:05:14.403 DEBUG:teuthology.orchestra.run.smithi047:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:05:14.405 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:05:14.423 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-29T11:05:14.430 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-29T11:05:14.439 INFO:teuthology.task.internal:Creating archive directory... 2024-03-29T11:05:14.439 DEBUG:teuthology.orchestra.run.smithi047:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:05:14.462 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:05:14.606 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-29T11:05:14.613 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-29T11:05:14.613 DEBUG:teuthology.orchestra.run.smithi047:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:05:14.615 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:05:14.650 INFO:teuthology.orchestra.run.smithi047.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:05:14.653 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:05:14.664 INFO:teuthology.orchestra.run.smithi047.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:05:14.668 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:05:14.669 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-29T11:05:14.676 INFO:teuthology.task.internal:Configuring sudo... 2024-03-29T11:05:14.676 DEBUG:teuthology.orchestra.run.smithi047:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:05:14.709 DEBUG:teuthology.orchestra.run.smithi121:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:05:14.739 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-29T11:05:14.748 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-29T11:05:14.748 DEBUG:teuthology.orchestra.run.smithi047:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:05:14.779 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:05:14.796 DEBUG:teuthology.orchestra.run.smithi047:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:05:14.875 DEBUG:teuthology.orchestra.run.smithi047:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:05:14.937 DEBUG:teuthology.orchestra.run.smithi047:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:05:14.977 DEBUG:teuthology.orchestra.run.smithi047:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:05:15.024 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-29T11:05:15.024 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:05:15.094 DEBUG:teuthology.orchestra.run.smithi121:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:05:15.139 DEBUG:teuthology.orchestra.run.smithi121:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:05:15.208 DEBUG:teuthology.orchestra.run.smithi121:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:05:15.252 DEBUG:teuthology.orchestra.run.smithi121:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:05:15.296 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-29T11:05:15.297 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:05:15.366 DEBUG:teuthology.orchestra.run.smithi047:> sudo service rsyslog restart 2024-03-29T11:05:15.368 DEBUG:teuthology.orchestra.run.smithi121:> sudo service rsyslog restart 2024-03-29T11:05:15.409 INFO:teuthology.orchestra.run.smithi047.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:05:15.458 INFO:teuthology.orchestra.run.smithi121.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:05:15.822 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-29T11:05:15.829 INFO:teuthology.task.internal:Starting timer... 2024-03-29T11:05:15.829 INFO:teuthology.run_tasks:Running task pcp... 2024-03-29T11:05:15.851 INFO:teuthology.run_tasks:Running task selinux... 2024-03-29T11:05:15.859 DEBUG:teuthology.orchestra.run.smithi047:> sudo service auditd rotate 2024-03-29T11:05:15.924 INFO:teuthology.orchestra.run.smithi047.stdout:Rotating logs: 2024-03-29T11:05:15.926 DEBUG:teuthology.orchestra.run.smithi121:> sudo service auditd rotate 2024-03-29T11:05:15.969 INFO:teuthology.orchestra.run.smithi121.stdout:Rotating logs: 2024-03-29T11:05:15.971 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-29T11:05:15.971 DEBUG:teuthology.orchestra.run.smithi047:> /usr/sbin/getenforce 2024-03-29T11:05:15.996 INFO:teuthology.orchestra.run.smithi047.stdout:Permissive 2024-03-29T11:05:15.997 DEBUG:teuthology.orchestra.run.smithi121:> /usr/sbin/getenforce 2024-03-29T11:05:16.032 INFO:teuthology.orchestra.run.smithi121.stdout:Permissive 2024-03-29T11:05:16.032 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi047.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi121.front.sepia.ceph.com': 'permissive'} 2024-03-29T11:05:16.033 DEBUG:teuthology.orchestra.run.smithi047:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:05:16.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:05:16.066 DEBUG:teuthology.orchestra.run.smithi121:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:05:16.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:05:16.100 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-29T11:05:16.100 DEBUG:teuthology.orchestra.run.smithi047:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:05:16.140 DEBUG:teuthology.orchestra.run.smithi121:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:05:16.174 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-29T11:05:16.183 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-29T11:05:16.188 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-29T11:05:16.313 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-29T11:05:16.331 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-29T11:05:16.332 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi047.front.sepia.ceph.com,smithi121.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-29T11:09:29.377 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi047.front.sepia.ceph.com'), Remote(name='ubuntu@smithi121.front.sepia.ceph.com')] 2024-03-29T11:09:29.378 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi047.front.sepia.ceph.com' 2024-03-29T11:09:29.380 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:09:29.472 DEBUG:teuthology.orchestra.run.smithi047:> true 2024-03-29T11:09:29.549 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi047.front.sepia.ceph.com' 2024-03-29T11:09:29.549 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-03-29T11:09:29.550 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:09:29.638 DEBUG:teuthology.orchestra.run.smithi121:> true 2024-03-29T11:09:29.713 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-03-29T11:09:29.714 INFO:teuthology.run_tasks:Running task clock... 2024-03-29T11:09:29.724 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-29T11:09:29.724 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:09:29.725 DEBUG:teuthology.orchestra.run.smithi047:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:09:29.727 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:09:29.728 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:09:29.757 INFO:teuthology.orchestra.run.smithi047.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:09:29.774 INFO:teuthology.orchestra.run.smithi047.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:09:29.783 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:09:29.799 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:09:29.803 INFO:teuthology.orchestra.run.smithi047.stderr:sudo: ntpd: command not found 2024-03-29T11:09:29.818 INFO:teuthology.orchestra.run.smithi047.stdout:506 Cannot talk to daemon 2024-03-29T11:09:29.828 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: ntpd: command not found 2024-03-29T11:09:29.833 INFO:teuthology.orchestra.run.smithi047.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:09:29.841 INFO:teuthology.orchestra.run.smithi121.stdout:506 Cannot talk to daemon 2024-03-29T11:09:29.850 INFO:teuthology.orchestra.run.smithi047.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:09:29.857 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:09:29.873 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:09:29.901 INFO:teuthology.orchestra.run.smithi047.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:09:29.904 INFO:teuthology.orchestra.run.smithi047.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:09:29.904 INFO:teuthology.orchestra.run.smithi047.stdout:=============================================================================== 2024-03-29T11:09:29.904 INFO:teuthology.orchestra.run.smithi047.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:09:29.904 INFO:teuthology.orchestra.run.smithi047.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:09:29.904 INFO:teuthology.orchestra.run.smithi047.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:09:29.904 INFO:teuthology.orchestra.run.smithi047.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:09:29.925 INFO:teuthology.orchestra.run.smithi121.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:09:29.928 INFO:teuthology.orchestra.run.smithi121.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:09:29.928 INFO:teuthology.orchestra.run.smithi121.stdout:=============================================================================== 2024-03-29T11:09:29.928 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:09:29.928 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:09:29.929 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:09:29.929 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:09:29.930 INFO:teuthology.run_tasks:Running task install... 2024-03-29T11:09:29.937 DEBUG:teuthology.task.install:project ceph 2024-03-29T11:09:29.938 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-29T11:09:29.938 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-29T11:09:29.938 INFO:teuthology.task.install:Using flavor: default 2024-03-29T11:09:29.943 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-29T11:09:29.944 INFO:teuthology.task.install:extra packages: [] 2024-03-29T11:09:29.944 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:09:29.944 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:09:29.946 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:09:29.946 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:09:30.096 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:09:30.096 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:09:30.161 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:09:30.161 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:09:30.266 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:09:30.266 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-29T11:09:30.266 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:09:30.294 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:09:30.295 DEBUG:teuthology.orchestra.run.smithi121:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:09:30.331 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:09:30.331 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-29T11:09:30.332 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:09:30.360 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:09:30.360 DEBUG:teuthology.orchestra.run.smithi047:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:09:30.361 DEBUG:teuthology.orchestra.run.smithi121:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:09:30.428 DEBUG:teuthology.orchestra.run.smithi047:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:09:30.441 DEBUG:teuthology.orchestra.run.smithi121:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:09:30.507 INFO:teuthology.orchestra.run.smithi121.stdout:check_obsoletes = 1 2024-03-29T11:09:30.508 DEBUG:teuthology.orchestra.run.smithi047:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:09:30.509 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum clean all 2024-03-29T11:09:30.574 INFO:teuthology.orchestra.run.smithi047.stdout:check_obsoletes = 1 2024-03-29T11:09:30.575 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum clean all 2024-03-29T11:09:30.757 INFO:teuthology.orchestra.run.smithi121.stdout:74 files removed 2024-03-29T11:09:30.783 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-radosgw 2024-03-29T11:09:30.829 INFO:teuthology.orchestra.run.smithi047.stdout:74 files removed 2024-03-29T11:09:30.855 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-radosgw 2024-03-29T11:09:31.411 INFO:teuthology.orchestra.run.smithi121.stdout:ceph packages for x86_64 248 kB/s | 82 kB 00:00 2024-03-29T11:09:31.516 INFO:teuthology.orchestra.run.smithi047.stdout:ceph packages for x86_64 231 kB/s | 82 kB 00:00 2024-03-29T11:09:31.745 INFO:teuthology.orchestra.run.smithi121.stdout:ceph noarch packages 35 kB/s | 10 kB 00:00 2024-03-29T11:09:32.025 INFO:teuthology.orchestra.run.smithi047.stdout:ceph noarch packages 22 kB/s | 10 kB 00:00 2024-03-29T11:09:32.220 INFO:teuthology.orchestra.run.smithi121.stdout:ceph source packages 4.3 kB/s | 1.9 kB 00:00 2024-03-29T11:09:32.404 INFO:teuthology.orchestra.run.smithi047.stdout:ceph source packages 5.7 kB/s | 1.9 kB 00:00 2024-03-29T11:09:32.878 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-29T11:09:32.923 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS-9 - AppStream 10 MB/s | 6.7 MB 00:00 2024-03-29T11:09:34.390 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS-9 - BaseOS 5.6 MB/s | 2.0 MB 00:00 2024-03-29T11:09:34.412 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS-9 - BaseOS 5.9 MB/s | 2.0 MB 00:00 2024-03-29T11:09:36.136 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS Stream 9 - BaseOS 5.8 MB/s | 8.0 MB 00:01 2024-03-29T11:09:37.920 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS Stream 9 - AppStream 29 MB/s | 19 MB 00:00 2024-03-29T11:09:42.229 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.1 MB 00:00 2024-03-29T11:09:43.639 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - BaseOS 922 kB/s | 8.0 MB 00:08 2024-03-29T11:09:43.663 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS Stream 9 - Extras packages 83 kB/s | 16 kB 00:00 2024-03-29T11:09:44.576 INFO:teuthology.orchestra.run.smithi047.stdout:Extra Packages for Enterprise Linux 27 MB/s | 21 MB 00:00 2024-03-29T11:09:45.676 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-03-29T11:09:49.691 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.1 MB 00:00 2024-03-29T11:09:49.815 INFO:teuthology.orchestra.run.smithi047.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-03-29T11:09:51.275 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - Extras packages 37 kB/s | 16 kB 00:00 2024-03-29T11:09:52.516 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-29T11:09:52.536 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-29T11:09:52.536 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2024-03-29T11:09:52.536 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-29T11:09:52.537 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-29T11:09:52.537 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:09:52.537 INFO:teuthology.orchestra.run.smithi047.stdout:Upgrading: 2024-03-29T11:09:52.537 INFO:teuthology.orchestra.run.smithi047.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:09:52.537 INFO:teuthology.orchestra.run.smithi047.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:09:52.537 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2024-03-29T11:09:52.537 INFO:teuthology.orchestra.run.smithi047.stdout: boost-program-options 2024-03-29T11:09:52.537 INFO:teuthology.orchestra.run.smithi047.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:09:52.537 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:09:52.537 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:09:52.537 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:09:52.537 INFO:teuthology.orchestra.run.smithi047.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:09:52.538 INFO:teuthology.orchestra.run.smithi047.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:09:52.538 INFO:teuthology.orchestra.run.smithi047.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:09:52.538 INFO:teuthology.orchestra.run.smithi047.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:09:52.538 INFO:teuthology.orchestra.run.smithi047.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:09:52.538 INFO:teuthology.orchestra.run.smithi047.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:09:52.538 INFO:teuthology.orchestra.run.smithi047.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:09:52.538 INFO:teuthology.orchestra.run.smithi047.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:09:52.538 INFO:teuthology.orchestra.run.smithi047.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:09:52.538 INFO:teuthology.orchestra.run.smithi047.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:09:52.538 INFO:teuthology.orchestra.run.smithi047.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:09:52.538 INFO:teuthology.orchestra.run.smithi047.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:09:52.538 INFO:teuthology.orchestra.run.smithi047.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:09:52.538 INFO:teuthology.orchestra.run.smithi047.stdout: python3-ceph-argparse 2024-03-29T11:09:52.538 INFO:teuthology.orchestra.run.smithi047.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:09:52.539 INFO:teuthology.orchestra.run.smithi047.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:09:52.539 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:09:52.539 INFO:teuthology.orchestra.run.smithi047.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:09:52.539 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:09:52.539 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:09:52.539 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:09:52.539 INFO:teuthology.orchestra.run.smithi047.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:09:52.539 INFO:teuthology.orchestra.run.smithi047.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:09:52.539 INFO:teuthology.orchestra.run.smithi047.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:09:52.539 INFO:teuthology.orchestra.run.smithi047.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:09:52.539 INFO:teuthology.orchestra.run.smithi047.stdout:Installing weak dependencies: 2024-03-29T11:09:52.539 INFO:teuthology.orchestra.run.smithi047.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:09:52.539 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:09:52.539 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-29T11:09:52.540 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-29T11:09:52.540 INFO:teuthology.orchestra.run.smithi047.stdout:Install 30 Packages 2024-03-29T11:09:52.540 INFO:teuthology.orchestra.run.smithi047.stdout:Upgrade 2 Packages 2024-03-29T11:09:52.540 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:09:52.540 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 58 M 2024-03-29T11:09:52.540 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-29T11:09:52.626 INFO:teuthology.orchestra.run.smithi121.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-03-29T11:09:53.275 INFO:teuthology.orchestra.run.smithi047.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 10 MB/s | 5.3 MB 00:00 2024-03-29T11:09:53.316 INFO:teuthology.orchestra.run.smithi047.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 606 kB/s | 25 kB 00:00 2024-03-29T11:09:53.383 INFO:teuthology.orchestra.run.smithi047.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 11 MB/s | 742 kB 00:00 2024-03-29T11:09:53.450 INFO:teuthology.orchestra.run.smithi047.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 7.3 MB/s | 498 kB 00:00 2024-03-29T11:09:53.717 INFO:teuthology.orchestra.run.smithi047.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 19 MB/s | 5.1 MB 00:00 2024-03-29T11:09:53.809 INFO:teuthology.orchestra.run.smithi047.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 494 kB/s | 45 kB 00:00 2024-03-29T11:09:53.851 INFO:teuthology.orchestra.run.smithi047.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 3.1 MB/s | 131 kB 00:00 2024-03-29T11:09:53.893 INFO:teuthology.orchestra.run.smithi047.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 3.8 MB/s | 162 kB 00:00 2024-03-29T11:09:53.943 INFO:teuthology.orchestra.run.smithi047.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 6.3 MB/s | 320 kB 00:00 2024-03-29T11:09:54.010 INFO:teuthology.orchestra.run.smithi047.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 4.4 MB/s | 298 kB 00:00 2024-03-29T11:09:54.052 INFO:teuthology.orchestra.run.smithi047.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 2.3 MB/s | 99 kB 00:00 2024-03-29T11:09:54.194 INFO:teuthology.orchestra.run.smithi047.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 748 kB/s | 106 kB 00:00 2024-03-29T11:09:54.294 INFO:teuthology.orchestra.run.smithi047.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.6 MB/s | 159 kB 00:00 2024-03-29T11:09:54.337 INFO:teuthology.orchestra.run.smithi047.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-29T11:09:54.412 INFO:teuthology.orchestra.run.smithi047.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-03-29T11:09:54.462 INFO:teuthology.orchestra.run.smithi047.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-03-29T11:09:54.496 INFO:teuthology.orchestra.run.smithi047.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-03-29T11:09:54.529 INFO:teuthology.orchestra.run.smithi047.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 6.5 MB/s | 220 kB 00:00 2024-03-29T11:09:54.554 INFO:teuthology.orchestra.run.smithi047.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 1.4 MB/s | 35 kB 00:00 2024-03-29T11:09:54.579 INFO:teuthology.orchestra.run.smithi047.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.8 MB/s | 46 kB 00:00 2024-03-29T11:09:54.788 INFO:teuthology.orchestra.run.smithi047.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.4 MB/s | 309 kB 00:00 2024-03-29T11:09:55.155 INFO:teuthology.orchestra.run.smithi047.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-03-29T11:09:55.189 INFO:teuthology.orchestra.run.smithi047.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 786 kB/s | 26 kB 00:00 2024-03-29T11:09:55.222 INFO:teuthology.orchestra.run.smithi047.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 1.4 MB/s | 49 kB 00:00 2024-03-29T11:09:55.256 INFO:teuthology.orchestra.run.smithi047.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 2.0 MB/s | 67 kB 00:00 2024-03-29T11:09:55.348 INFO:teuthology.orchestra.run.smithi047.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 8.9 MB/s | 838 kB 00:00 2024-03-29T11:09:55.398 INFO:teuthology.orchestra.run.smithi047.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2024-03-29T11:09:55.573 INFO:teuthology.orchestra.run.smithi047.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 9.1 MB/s | 1.6 MB 00:00 2024-03-29T11:09:55.849 INFO:teuthology.orchestra.run.smithi047.stdout:(29/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 12 MB/s | 3.4 MB 00:00 2024-03-29T11:09:56.033 INFO:teuthology.orchestra.run.smithi047.stdout:(30/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 17 MB/s | 3.1 MB 00:00 2024-03-29T11:09:57.166 INFO:teuthology.orchestra.run.smithi047.stdout:(31/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 2.2 MB/s | 9.7 MB 00:04 2024-03-29T11:09:57.651 INFO:teuthology.orchestra.run.smithi121.stdout:lab-extras 19 kB/s | 1.7 kB 00:00 2024-03-29T11:10:00.276 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-29T11:10:00.279 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:10:00.279 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-03-29T11:10:00.279 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:10:00.280 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-29T11:10:00.280 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:10:00.280 INFO:teuthology.orchestra.run.smithi121.stdout:Upgrading: 2024-03-29T11:10:00.280 INFO:teuthology.orchestra.run.smithi121.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:10:00.280 INFO:teuthology.orchestra.run.smithi121.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:10:00.280 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-03-29T11:10:00.280 INFO:teuthology.orchestra.run.smithi121.stdout: boost-program-options 2024-03-29T11:10:00.280 INFO:teuthology.orchestra.run.smithi121.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:10:00.280 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:10:00.280 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:10:00.280 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:10:00.280 INFO:teuthology.orchestra.run.smithi121.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:10:00.280 INFO:teuthology.orchestra.run.smithi121.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:10:00.281 INFO:teuthology.orchestra.run.smithi121.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:10:00.281 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:10:00.281 INFO:teuthology.orchestra.run.smithi121.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:10:00.281 INFO:teuthology.orchestra.run.smithi121.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:10:00.281 INFO:teuthology.orchestra.run.smithi121.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:10:00.281 INFO:teuthology.orchestra.run.smithi121.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:10:00.281 INFO:teuthology.orchestra.run.smithi121.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:10:00.281 INFO:teuthology.orchestra.run.smithi121.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:10:00.281 INFO:teuthology.orchestra.run.smithi121.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:10:00.281 INFO:teuthology.orchestra.run.smithi121.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:10:00.281 INFO:teuthology.orchestra.run.smithi121.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:10:00.281 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-argparse 2024-03-29T11:10:00.281 INFO:teuthology.orchestra.run.smithi121.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:10:00.281 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:10:00.282 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:10:00.282 INFO:teuthology.orchestra.run.smithi121.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:10:00.282 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:10:00.282 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:10:00.282 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:10:00.282 INFO:teuthology.orchestra.run.smithi121.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:10:00.282 INFO:teuthology.orchestra.run.smithi121.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:10:00.282 INFO:teuthology.orchestra.run.smithi121.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:10:00.282 INFO:teuthology.orchestra.run.smithi121.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:10:00.282 INFO:teuthology.orchestra.run.smithi121.stdout:Installing weak dependencies: 2024-03-29T11:10:00.282 INFO:teuthology.orchestra.run.smithi121.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:10:00.282 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:10:00.282 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-29T11:10:00.282 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:10:00.283 INFO:teuthology.orchestra.run.smithi121.stdout:Install 30 Packages 2024-03-29T11:10:00.283 INFO:teuthology.orchestra.run.smithi121.stdout:Upgrade 2 Packages 2024-03-29T11:10:00.283 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:10:00.283 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 58 M 2024-03-29T11:10:00.283 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-29T11:10:01.335 INFO:teuthology.orchestra.run.smithi121.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 7.8 MB/s | 5.3 MB 00:00 2024-03-29T11:10:01.377 INFO:teuthology.orchestra.run.smithi121.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 606 kB/s | 25 kB 00:00 2024-03-29T11:10:01.477 INFO:teuthology.orchestra.run.smithi121.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 7.3 MB/s | 742 kB 00:00 2024-03-29T11:10:01.544 INFO:teuthology.orchestra.run.smithi121.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 7.3 MB/s | 498 kB 00:00 2024-03-29T11:10:01.582 INFO:teuthology.orchestra.run.smithi047.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 2.3 MB/s | 20 MB 00:08 2024-03-29T11:10:01.587 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:10:01.587 INFO:teuthology.orchestra.run.smithi047.stdout:Total 6.4 MB/s | 58 MB 00:09 2024-03-29T11:10:01.587 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-29T11:10:01.630 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-29T11:10:01.630 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-29T11:10:02.019 INFO:teuthology.orchestra.run.smithi121.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 11 MB/s | 5.1 MB 00:00 2024-03-29T11:10:02.061 INFO:teuthology.orchestra.run.smithi121.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 1.1 MB/s | 45 kB 00:00 2024-03-29T11:10:02.116 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-29T11:10:02.117 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-29T11:10:02.120 INFO:teuthology.orchestra.run.smithi121.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 2.2 MB/s | 131 kB 00:00 2024-03-29T11:10:02.187 INFO:teuthology.orchestra.run.smithi121.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 2.4 MB/s | 162 kB 00:00 2024-03-29T11:10:02.256 INFO:teuthology.orchestra.run.smithi121.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 4.7 MB/s | 320 kB 00:00 2024-03-29T11:10:02.362 INFO:teuthology.orchestra.run.smithi121.stdout:(10/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 5.7 MB/s | 9.7 MB 00:01 2024-03-29T11:10:02.387 INFO:teuthology.orchestra.run.smithi121.stdout:(11/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 2.2 MB/s | 298 kB 00:00 2024-03-29T11:10:02.446 INFO:teuthology.orchestra.run.smithi121.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.2 MB/s | 99 kB 00:00 2024-03-29T11:10:02.513 INFO:teuthology.orchestra.run.smithi121.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.4 MB/s | 159 kB 00:00 2024-03-29T11:10:02.539 INFO:teuthology.orchestra.run.smithi121.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 703 kB/s | 106 kB 00:00 2024-03-29T11:10:02.555 INFO:teuthology.orchestra.run.smithi121.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-29T11:10:02.605 INFO:teuthology.orchestra.run.smithi121.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-03-29T11:10:02.631 INFO:teuthology.orchestra.run.smithi121.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-03-29T11:10:02.648 INFO:teuthology.orchestra.run.smithi121.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 993 kB/s | 41 kB 00:00 2024-03-29T11:10:02.673 INFO:teuthology.orchestra.run.smithi121.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 5.1 MB/s | 220 kB 00:00 2024-03-29T11:10:02.690 INFO:teuthology.orchestra.run.smithi121.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 832 kB/s | 35 kB 00:00 2024-03-29T11:10:02.709 INFO:teuthology.orchestra.run.smithi121.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.3 MB/s | 46 kB 00:00 2024-03-29T11:10:02.916 INFO:teuthology.orchestra.run.smithi121.stdout:(22/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 8.8 MB/s | 20 MB 00:02 2024-03-29T11:10:03.224 INFO:teuthology.orchestra.run.smithi121.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 100 kB/s | 26 kB 00:00 2024-03-29T11:10:03.308 INFO:teuthology.orchestra.run.smithi121.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 500 kB/s | 309 kB 00:00 2024-03-29T11:10:03.341 INFO:teuthology.orchestra.run.smithi121.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 415 kB/s | 49 kB 00:00 2024-03-29T11:10:03.383 INFO:teuthology.orchestra.run.smithi121.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 894 kB/s | 67 kB 00:00 2024-03-29T11:10:03.475 INFO:teuthology.orchestra.run.smithi121.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.0 MB/s | 192 kB 00:00 2024-03-29T11:10:03.542 INFO:teuthology.orchestra.run.smithi121.stdout:(28/32): libarrow-9.0.0-10.el9.x86_64.rpm 5.3 MB/s | 4.4 MB 00:00 2024-03-29T11:10:03.592 INFO:teuthology.orchestra.run.smithi121.stdout:(29/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.3 MB/s | 838 kB 00:00 2024-03-29T11:10:03.612 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-29T11:10:03.696 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:10:03.709 INFO:teuthology.orchestra.run.smithi121.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 6.8 MB/s | 1.6 MB 00:00 2024-03-29T11:10:03.737 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:10:03.771 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:10:03.802 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:10:03.819 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:10:03.835 INFO:teuthology.orchestra.run.smithi121.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 12 MB/s | 3.4 MB 00:00 2024-03-29T11:10:04.020 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:10:04.026 INFO:teuthology.orchestra.run.smithi121.stdout:(32/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 7.2 MB/s | 3.1 MB 00:00 2024-03-29T11:10:04.032 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:10:04.032 INFO:teuthology.orchestra.run.smithi121.stdout:Total 15 MB/s | 58 MB 00:03 2024-03-29T11:10:04.032 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-29T11:10:04.064 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-29T11:10:04.064 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-29T11:10:04.074 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:10:04.410 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:10:04.447 INFO:teuthology.orchestra.run.smithi047.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:10:04.539 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-29T11:10:04.539 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-29T11:10:04.539 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:10:04.629 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:10:04.657 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:10:04.703 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:10:04.724 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:10:04.768 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:10:04.837 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:10:04.860 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:10:04.907 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:10:04.941 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:10:05.052 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:10:05.084 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:10:05.125 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:10:05.157 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:10:05.186 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:10:05.219 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:10:05.558 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:10:05.590 INFO:teuthology.orchestra.run.smithi047.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:10:05.680 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:10:05.712 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:10:05.757 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:10:06.017 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-29T11:10:06.128 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:10:06.162 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:10:06.190 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:10:06.212 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:10:06.230 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:10:06.236 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:10:06.267 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:10:06.348 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:10:06.398 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:10:06.426 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:10:06.452 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:10:06.453 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:10:06.794 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:10:06.858 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:10:06.975 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:10:07.150 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:10:07.177 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:10:07.239 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:10:07.268 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:10:07.312 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:10:07.381 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:10:07.404 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:10:07.468 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:10:07.502 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:10:07.610 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:10:07.653 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:10:07.694 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:10:07.726 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:10:07.766 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:10:07.796 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:10:08.104 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:10:08.135 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:10:08.233 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:10:08.264 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:10:08.308 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:10:08.757 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:10:08.764 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:10:08.795 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:10:08.799 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:10:08.866 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:10:08.934 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:10:08.964 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:10:09.458 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:10:09.495 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:10:09.584 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:10:09.584 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:10:09.584 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:10:09.584 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:10:09.621 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:10:11.105 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:10:11.152 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:10:11.696 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:10:11.730 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:10:11.814 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:10:11.814 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:10:11.814 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:10:11.815 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:10:11.848 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:10:19.579 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:10:19.580 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /sys 2024-03-29T11:10:19.580 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /proc 2024-03-29T11:10:19.580 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /mnt 2024-03-29T11:10:19.580 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /var/tmp 2024-03-29T11:10:19.580 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /home 2024-03-29T11:10:19.580 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /root 2024-03-29T11:10:19.580 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /tmp 2024-03-29T11:10:19.580 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:10:20.504 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:10:20.528 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:10:20.528 INFO:teuthology.orchestra.run.smithi047.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:10:20.528 INFO:teuthology.orchestra.run.smithi047.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:10:20.528 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:10:20.528 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:10:20.528 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:10:20.530 INFO:teuthology.orchestra.run.smithi047.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:10:20.617 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:10:20.618 INFO:teuthology.orchestra.run.smithi047.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:10:21.580 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:10:21.580 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /sys 2024-03-29T11:10:21.580 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /proc 2024-03-29T11:10:21.580 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /mnt 2024-03-29T11:10:21.580 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /var/tmp 2024-03-29T11:10:21.580 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /home 2024-03-29T11:10:21.580 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /root 2024-03-29T11:10:21.580 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /tmp 2024-03-29T11:10:21.580 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:10:21.759 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:10:21.759 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:10:21.759 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:10:21.759 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:10:21.759 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:10:21.759 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:10:21.759 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:10:21.759 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:10:21.760 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:10:21.760 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:10:21.760 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:10:21.760 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:10:21.760 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:10:21.760 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:10:21.760 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:10:21.760 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:10:21.760 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:10:21.760 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:10:21.760 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:10:21.760 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:10:21.760 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:10:21.760 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:10:21.761 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:10:21.761 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:10:21.761 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:10:21.761 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:10:21.761 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:10:21.761 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:10:21.761 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:10:21.761 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:10:21.761 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:10:21.761 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:10:21.761 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:10:21.761 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:10:22.478 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:10:22.502 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:10:22.503 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:10:22.503 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:10:22.503 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:10:22.503 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:10:22.503 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:10:22.505 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:10:22.588 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:10:22.588 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:10:22.602 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:10:22.602 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:10:22.602 INFO:teuthology.orchestra.run.smithi047.stdout:Upgraded: 2024-03-29T11:10:22.602 INFO:teuthology.orchestra.run.smithi047.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:22.602 INFO:teuthology.orchestra.run.smithi047.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:22.602 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-29T11:10:22.602 INFO:teuthology.orchestra.run.smithi047.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:10:22.602 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:22.602 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:22.602 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:22.603 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:22.603 INFO:teuthology.orchestra.run.smithi047.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:10:22.603 INFO:teuthology.orchestra.run.smithi047.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:10:22.603 INFO:teuthology.orchestra.run.smithi047.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:10:22.603 INFO:teuthology.orchestra.run.smithi047.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:22.603 INFO:teuthology.orchestra.run.smithi047.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:10:22.603 INFO:teuthology.orchestra.run.smithi047.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:10:22.603 INFO:teuthology.orchestra.run.smithi047.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:10:22.603 INFO:teuthology.orchestra.run.smithi047.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:22.603 INFO:teuthology.orchestra.run.smithi047.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:10:22.603 INFO:teuthology.orchestra.run.smithi047.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:22.603 INFO:teuthology.orchestra.run.smithi047.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:10:22.603 INFO:teuthology.orchestra.run.smithi047.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:10:22.604 INFO:teuthology.orchestra.run.smithi047.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:10:22.604 INFO:teuthology.orchestra.run.smithi047.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:22.604 INFO:teuthology.orchestra.run.smithi047.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:22.604 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:22.604 INFO:teuthology.orchestra.run.smithi047.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:10:22.604 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:22.604 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:22.604 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:22.604 INFO:teuthology.orchestra.run.smithi047.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:10:22.604 INFO:teuthology.orchestra.run.smithi047.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:10:22.604 INFO:teuthology.orchestra.run.smithi047.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:10:22.604 INFO:teuthology.orchestra.run.smithi047.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:10:22.604 INFO:teuthology.orchestra.run.smithi047.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:10:22.604 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:10:22.605 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-29T11:10:22.827 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-test 2024-03-29T11:10:23.395 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:00:34 ago on Fri 29 Mar 2024 11:09:49 AM UTC. 2024-03-29T11:10:23.561 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-29T11:10:23.562 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-29T11:10:23.562 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2024-03-29T11:10:23.562 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-29T11:10:23.562 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-29T11:10:23.562 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:10:23.562 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2024-03-29T11:10:23.562 INFO:teuthology.orchestra.run.smithi047.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:10:23.563 INFO:teuthology.orchestra.run.smithi047.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:10:23.563 INFO:teuthology.orchestra.run.smithi047.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:10:23.563 INFO:teuthology.orchestra.run.smithi047.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:10:23.563 INFO:teuthology.orchestra.run.smithi047.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:10:23.563 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:10:23.563 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-29T11:10:23.563 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-29T11:10:23.563 INFO:teuthology.orchestra.run.smithi047.stdout:Install 6 Packages 2024-03-29T11:10:23.563 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:10:23.564 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 48 M 2024-03-29T11:10:23.564 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 204 M 2024-03-29T11:10:23.564 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-29T11:10:23.674 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:10:23.674 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:10:23.675 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:10:23.675 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:10:23.675 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:10:23.675 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:10:23.675 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:10:23.675 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:10:23.675 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:10:23.675 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:10:23.675 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:10:23.675 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:10:23.675 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:10:23.675 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:10:23.676 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:10:23.676 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:10:23.676 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:10:23.676 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:10:23.676 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:10:23.676 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:10:23.676 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:10:23.676 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:10:23.676 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:10:23.676 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:10:23.676 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:10:23.676 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:10:23.676 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:10:23.677 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:10:23.677 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:10:23.677 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:10:23.677 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:10:23.677 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:10:23.677 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:10:23.677 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:10:24.004 INFO:teuthology.orchestra.run.smithi047.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.2 MB/s | 217 kB 00:00 2024-03-29T11:10:24.046 INFO:teuthology.orchestra.run.smithi047.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.1 MB/s | 300 kB 00:00 2024-03-29T11:10:24.080 INFO:teuthology.orchestra.run.smithi047.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-29T11:10:24.105 INFO:teuthology.orchestra.run.smithi047.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 602 kB/s | 163 kB 00:00 2024-03-29T11:10:24.224 INFO:teuthology.orchestra.run.smithi047.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 1.3 MB/s | 187 kB 00:00 2024-03-29T11:10:24.729 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:10:24.729 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:10:24.730 INFO:teuthology.orchestra.run.smithi121.stdout:Upgraded: 2024-03-29T11:10:24.730 INFO:teuthology.orchestra.run.smithi121.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:24.730 INFO:teuthology.orchestra.run.smithi121.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:24.730 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-29T11:10:24.730 INFO:teuthology.orchestra.run.smithi121.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:10:24.730 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:24.730 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:24.730 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:24.730 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:24.730 INFO:teuthology.orchestra.run.smithi121.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:10:24.730 INFO:teuthology.orchestra.run.smithi121.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:10:24.730 INFO:teuthology.orchestra.run.smithi121.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:10:24.731 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:24.731 INFO:teuthology.orchestra.run.smithi121.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:10:24.731 INFO:teuthology.orchestra.run.smithi121.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:10:24.731 INFO:teuthology.orchestra.run.smithi121.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:10:24.731 INFO:teuthology.orchestra.run.smithi121.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:24.731 INFO:teuthology.orchestra.run.smithi121.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:10:24.731 INFO:teuthology.orchestra.run.smithi121.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:24.731 INFO:teuthology.orchestra.run.smithi121.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:10:24.731 INFO:teuthology.orchestra.run.smithi121.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:10:24.731 INFO:teuthology.orchestra.run.smithi121.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:10:24.731 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:24.731 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:24.731 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:24.731 INFO:teuthology.orchestra.run.smithi121.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:10:24.732 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:24.732 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:24.732 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:24.732 INFO:teuthology.orchestra.run.smithi121.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:10:24.732 INFO:teuthology.orchestra.run.smithi121.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:10:24.732 INFO:teuthology.orchestra.run.smithi121.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:10:24.732 INFO:teuthology.orchestra.run.smithi121.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:10:24.732 INFO:teuthology.orchestra.run.smithi121.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:10:24.732 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:10:24.732 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-29T11:10:25.029 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-test 2024-03-29T11:10:25.589 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:28 ago on Fri 29 Mar 2024 11:09:57 AM UTC. 2024-03-29T11:10:25.738 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-29T11:10:25.738 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:10:25.738 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-03-29T11:10:25.739 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:10:25.739 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-29T11:10:25.739 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:10:25.739 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-03-29T11:10:25.739 INFO:teuthology.orchestra.run.smithi121.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:10:25.739 INFO:teuthology.orchestra.run.smithi121.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:10:25.739 INFO:teuthology.orchestra.run.smithi121.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:10:25.739 INFO:teuthology.orchestra.run.smithi121.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:10:25.739 INFO:teuthology.orchestra.run.smithi121.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:10:25.739 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:10:25.739 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-29T11:10:25.739 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:10:25.739 INFO:teuthology.orchestra.run.smithi121.stdout:Install 6 Packages 2024-03-29T11:10:25.739 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:10:25.740 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 48 M 2024-03-29T11:10:25.740 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 204 M 2024-03-29T11:10:25.740 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-29T11:10:25.942 INFO:teuthology.orchestra.run.smithi121.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-29T11:10:25.993 INFO:teuthology.orchestra.run.smithi121.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-29T11:10:26.026 INFO:teuthology.orchestra.run.smithi121.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-29T11:10:26.051 INFO:teuthology.orchestra.run.smithi121.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 595 kB/s | 163 kB 00:00 2024-03-29T11:10:26.189 INFO:teuthology.orchestra.run.smithi047.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 20 MB/s | 47 MB 00:02 2024-03-29T11:10:26.192 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:10:26.192 INFO:teuthology.orchestra.run.smithi047.stdout:Total 18 MB/s | 48 MB 00:02 2024-03-29T11:10:26.193 INFO:teuthology.orchestra.run.smithi121.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 1.1 MB/s | 187 kB 00:00 2024-03-29T11:10:26.207 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-29T11:10:26.230 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-29T11:10:26.230 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-29T11:10:26.621 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-29T11:10:26.622 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-29T11:10:27.218 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-29T11:10:27.299 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:10:27.344 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:10:27.376 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:10:27.417 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:10:27.442 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:10:28.144 INFO:teuthology.orchestra.run.smithi121.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 20 MB/s | 47 MB 00:02 2024-03-29T11:10:28.147 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:10:28.147 INFO:teuthology.orchestra.run.smithi121.stdout:Total 20 MB/s | 48 MB 00:02 2024-03-29T11:10:28.162 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-29T11:10:28.183 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-29T11:10:28.183 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-29T11:10:28.560 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-29T11:10:28.560 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-29T11:10:29.155 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-29T11:10:29.219 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:10:29.266 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:10:29.306 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:10:29.344 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:10:29.372 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:10:32.166 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:10:32.261 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:10:33.852 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:10:33.937 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:10:34.611 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:10:34.611 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:10:34.611 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:10:34.612 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:10:34.612 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:10:34.612 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:10:35.049 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:10:35.049 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:10:35.049 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-29T11:10:35.049 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:35.049 INFO:teuthology.orchestra.run.smithi047.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:10:35.049 INFO:teuthology.orchestra.run.smithi047.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:35.049 INFO:teuthology.orchestra.run.smithi047.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:10:35.049 INFO:teuthology.orchestra.run.smithi047.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:10:35.049 INFO:teuthology.orchestra.run.smithi047.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:10:35.049 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:10:35.049 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-29T11:10:35.367 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph 2024-03-29T11:10:35.955 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:00:46 ago on Fri 29 Mar 2024 11:09:49 AM UTC. 2024-03-29T11:10:36.118 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-29T11:10:36.120 INFO:teuthology.orchestra.run.smithi047.stdout:========================================================================================= 2024-03-29T11:10:36.121 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2024-03-29T11:10:36.121 INFO:teuthology.orchestra.run.smithi047.stdout:========================================================================================= 2024-03-29T11:10:36.121 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-29T11:10:36.121 INFO:teuthology.orchestra.run.smithi047.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:10:36.121 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2024-03-29T11:10:36.121 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:10:36.121 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:10:36.121 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:10:36.121 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:10:36.121 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:10:36.121 INFO:teuthology.orchestra.run.smithi047.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:10:36.121 INFO:teuthology.orchestra.run.smithi047.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:10:36.121 INFO:teuthology.orchestra.run.smithi047.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:10:36.122 INFO:teuthology.orchestra.run.smithi047.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:10:36.122 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:10:36.122 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:10:36.122 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:10:36.122 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:10:36.122 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:10:36.122 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:10:36.122 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:10:36.122 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:10:36.122 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:10:36.122 INFO:teuthology.orchestra.run.smithi047.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:10:36.122 INFO:teuthology.orchestra.run.smithi047.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:10:36.122 INFO:teuthology.orchestra.run.smithi047.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:10:36.123 INFO:teuthology.orchestra.run.smithi047.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:10:36.123 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:10:36.123 INFO:teuthology.orchestra.run.smithi047.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:10:36.123 INFO:teuthology.orchestra.run.smithi047.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:10:36.123 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:10:36.123 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:10:36.123 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:10:36.123 INFO:teuthology.orchestra.run.smithi047.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:10:36.123 INFO:teuthology.orchestra.run.smithi047.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:10:36.123 INFO:teuthology.orchestra.run.smithi047.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:10:36.123 INFO:teuthology.orchestra.run.smithi047.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:10:36.123 INFO:teuthology.orchestra.run.smithi047.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:10:36.123 INFO:teuthology.orchestra.run.smithi047.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:10:36.123 INFO:teuthology.orchestra.run.smithi047.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:10:36.124 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:10:36.124 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-29T11:10:36.124 INFO:teuthology.orchestra.run.smithi047.stdout:========================================================================================= 2024-03-29T11:10:36.124 INFO:teuthology.orchestra.run.smithi047.stdout:Install 36 Packages 2024-03-29T11:10:36.124 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:10:36.124 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 30 M 2024-03-29T11:10:36.124 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 107 M 2024-03-29T11:10:36.125 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-29T11:10:36.365 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:10:36.366 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:10:36.366 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:10:36.366 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:10:36.366 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:10:36.366 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:10:36.779 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:10:36.779 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:10:36.779 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-29T11:10:36.779 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:36.780 INFO:teuthology.orchestra.run.smithi121.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:10:36.780 INFO:teuthology.orchestra.run.smithi121.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:36.780 INFO:teuthology.orchestra.run.smithi121.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:10:36.780 INFO:teuthology.orchestra.run.smithi121.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:10:36.780 INFO:teuthology.orchestra.run.smithi121.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:10:36.780 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:10:36.780 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-29T11:10:36.793 INFO:teuthology.orchestra.run.smithi047.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 40 kB/s | 6.6 kB 00:00 2024-03-29T11:10:37.018 INFO:teuthology.orchestra.run.smithi047.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 3.8 MB/s | 1.5 MB 00:00 2024-03-29T11:10:37.060 INFO:teuthology.orchestra.run.smithi047.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 4.9 MB/s | 2.1 MB 00:00 2024-03-29T11:10:37.113 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph 2024-03-29T11:10:37.143 INFO:teuthology.orchestra.run.smithi047.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 2.9 MB/s | 247 kB 00:00 2024-03-29T11:10:37.269 INFO:teuthology.orchestra.run.smithi047.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-03-29T11:10:37.310 INFO:teuthology.orchestra.run.smithi047.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-29T11:10:37.344 INFO:teuthology.orchestra.run.smithi047.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 945 kB/s | 32 kB 00:00 2024-03-29T11:10:37.377 INFO:teuthology.orchestra.run.smithi047.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-29T11:10:37.411 INFO:teuthology.orchestra.run.smithi047.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-29T11:10:37.453 INFO:teuthology.orchestra.run.smithi047.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-29T11:10:37.520 INFO:teuthology.orchestra.run.smithi047.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-29T11:10:37.554 INFO:teuthology.orchestra.run.smithi047.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-29T11:10:37.587 INFO:teuthology.orchestra.run.smithi047.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-29T11:10:37.621 INFO:teuthology.orchestra.run.smithi047.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-03-29T11:10:37.705 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:40 ago on Fri 29 Mar 2024 11:09:57 AM UTC. 2024-03-29T11:10:37.872 INFO:teuthology.orchestra.run.smithi047.stdout:(15/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 20 MB/s | 17 MB 00:00 2024-03-29T11:10:37.875 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-29T11:10:37.877 INFO:teuthology.orchestra.run.smithi121.stdout:========================================================================================= 2024-03-29T11:10:37.877 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-03-29T11:10:37.877 INFO:teuthology.orchestra.run.smithi121.stdout:========================================================================================= 2024-03-29T11:10:37.877 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-29T11:10:37.878 INFO:teuthology.orchestra.run.smithi121.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:10:37.878 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-03-29T11:10:37.878 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:10:37.878 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:10:37.878 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:10:37.878 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:10:37.878 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:10:37.878 INFO:teuthology.orchestra.run.smithi121.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:10:37.878 INFO:teuthology.orchestra.run.smithi121.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:10:37.878 INFO:teuthology.orchestra.run.smithi121.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:10:37.878 INFO:teuthology.orchestra.run.smithi121.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:10:37.878 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:10:37.878 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:10:37.879 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:10:37.879 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:10:37.882 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:10:37.882 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:10:37.882 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:10:37.882 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:10:37.882 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:10:37.882 INFO:teuthology.orchestra.run.smithi121.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:10:37.882 INFO:teuthology.orchestra.run.smithi121.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:10:37.882 INFO:teuthology.orchestra.run.smithi121.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:10:37.882 INFO:teuthology.orchestra.run.smithi121.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:10:37.882 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:10:37.882 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:10:37.883 INFO:teuthology.orchestra.run.smithi121.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:10:37.883 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:10:37.883 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:10:37.883 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:10:37.883 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:10:37.883 INFO:teuthology.orchestra.run.smithi121.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:10:37.883 INFO:teuthology.orchestra.run.smithi121.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:10:37.883 INFO:teuthology.orchestra.run.smithi121.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:10:37.883 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:10:37.883 INFO:teuthology.orchestra.run.smithi121.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:10:37.883 INFO:teuthology.orchestra.run.smithi121.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:10:37.883 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:10:37.883 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-29T11:10:37.883 INFO:teuthology.orchestra.run.smithi121.stdout:========================================================================================= 2024-03-29T11:10:37.884 INFO:teuthology.orchestra.run.smithi121.stdout:Install 36 Packages 2024-03-29T11:10:37.884 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:10:37.884 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 30 M 2024-03-29T11:10:37.884 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 107 M 2024-03-29T11:10:37.884 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-29T11:10:37.980 INFO:teuthology.orchestra.run.smithi047.stdout:(16/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 3.9 MB/s | 4.7 MB 00:01 2024-03-29T11:10:38.005 INFO:teuthology.orchestra.run.smithi047.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 167 kB/s | 22 kB 00:00 2024-03-29T11:10:38.214 INFO:teuthology.orchestra.run.smithi047.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 208 kB/s | 43 kB 00:00 2024-03-29T11:10:38.281 INFO:teuthology.orchestra.run.smithi047.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 502 kB/s | 151 kB 00:00 2024-03-29T11:10:38.332 INFO:teuthology.orchestra.run.smithi121.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 38 kB/s | 6.6 kB 00:00 2024-03-29T11:10:38.348 INFO:teuthology.orchestra.run.smithi047.stdout:(20/36): python3-requests-2.25.1-8.el9.noarch.r 172 kB/s | 125 kB 00:00 2024-03-29T11:10:38.423 INFO:teuthology.orchestra.run.smithi047.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 821 kB/s | 172 kB 00:00 2024-03-29T11:10:38.490 INFO:teuthology.orchestra.run.smithi047.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 75 kB/s | 11 kB 00:00 2024-03-29T11:10:38.524 INFO:teuthology.orchestra.run.smithi047.stdout:(23/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.4 MB/s | 358 kB 00:00 2024-03-29T11:10:38.541 INFO:teuthology.orchestra.run.smithi047.stdout:(24/36): python3-jaraco-classes-3.2.1-5.el9.noa 151 kB/s | 18 kB 00:00 2024-03-29T11:10:38.582 INFO:teuthology.orchestra.run.smithi121.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 3.5 MB/s | 1.5 MB 00:00 2024-03-29T11:10:38.583 INFO:teuthology.orchestra.run.smithi047.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 251 kB/s | 23 kB 00:00 2024-03-29T11:10:38.625 INFO:teuthology.orchestra.run.smithi047.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 235 kB/s | 20 kB 00:00 2024-03-29T11:10:38.641 INFO:teuthology.orchestra.run.smithi047.stdout:(27/36): python3-jaraco-functools-3.5.0-2.el9.n 166 kB/s | 19 kB 00:00 2024-03-29T11:10:38.700 INFO:teuthology.orchestra.run.smithi047.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 395 kB/s | 46 kB 00:00 2024-03-29T11:10:38.750 INFO:teuthology.orchestra.run.smithi047.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 628 kB/s | 79 kB 00:00 2024-03-29T11:10:38.801 INFO:teuthology.orchestra.run.smithi047.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 164 kB/s | 16 kB 00:00 2024-03-29T11:10:38.893 INFO:teuthology.orchestra.run.smithi047.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.1 MB/s | 272 kB 00:00 2024-03-29T11:10:38.899 INFO:teuthology.orchestra.run.smithi121.stdout:(3/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86_ 8.3 MB/s | 4.7 MB 00:00 2024-03-29T11:10:38.910 INFO:teuthology.orchestra.run.smithi047.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 566 kB/s | 90 kB 00:00 2024-03-29T11:10:38.935 INFO:teuthology.orchestra.run.smithi047.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 267 kB/s | 36 kB 00:00 2024-03-29T11:10:38.958 INFO:teuthology.orchestra.run.smithi121.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 4.1 MB/s | 247 kB 00:00 2024-03-29T11:10:39.060 INFO:teuthology.orchestra.run.smithi047.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.3 MB/s | 230 kB 00:00 2024-03-29T11:10:39.102 INFO:teuthology.orchestra.run.smithi047.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 120 kB/s | 20 kB 00:00 2024-03-29T11:10:39.142 INFO:teuthology.orchestra.run.smithi121.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2024-03-29T11:10:39.202 INFO:teuthology.orchestra.run.smithi047.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.4 MB/s | 427 kB 00:00 2024-03-29T11:10:39.206 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:10:39.206 INFO:teuthology.orchestra.run.smithi047.stdout:Total 9.6 MB/s | 30 MB 00:03 2024-03-29T11:10:39.208 INFO:teuthology.orchestra.run.smithi121.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.1 MB/s | 144 kB 00:00 2024-03-29T11:10:39.231 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-29T11:10:39.233 INFO:teuthology.orchestra.run.smithi121.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2024-03-29T11:10:39.259 INFO:teuthology.orchestra.run.smithi121.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.8 MB/s | 47 kB 00:00 2024-03-29T11:10:39.277 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-29T11:10:39.277 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-29T11:10:39.292 INFO:teuthology.orchestra.run.smithi121.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-29T11:10:39.351 INFO:teuthology.orchestra.run.smithi121.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-03-29T11:10:39.418 INFO:teuthology.orchestra.run.smithi121.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-29T11:10:39.577 INFO:teuthology.orchestra.run.smithi121.stdout:(12/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 17 MB/s | 17 MB 00:00 2024-03-29T11:10:39.602 INFO:teuthology.orchestra.run.smithi121.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 560 kB/s | 103 kB 00:00 2024-03-29T11:10:39.627 INFO:teuthology.orchestra.run.smithi121.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-29T11:10:39.652 INFO:teuthology.orchestra.run.smithi121.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-29T11:10:39.694 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-29T11:10:39.694 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-29T11:10:39.695 INFO:teuthology.orchestra.run.smithi121.stdout:(16/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86 1.4 MB/s | 2.1 MB 00:01 2024-03-29T11:10:39.870 INFO:teuthology.orchestra.run.smithi121.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 103 kB/s | 22 kB 00:00 2024-03-29T11:10:39.895 INFO:teuthology.orchestra.run.smithi121.stdout:(18/36): python3-requests-2.25.1-8.el9.noarch.r 467 kB/s | 125 kB 00:00 2024-03-29T11:10:40.045 INFO:teuthology.orchestra.run.smithi121.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 431 kB/s | 151 kB 00:00 2024-03-29T11:10:40.112 INFO:teuthology.orchestra.run.smithi121.stdout:(20/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 180 kB/s | 43 kB 00:00 2024-03-29T11:10:40.187 INFO:teuthology.orchestra.run.smithi121.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 142 kB/s | 11 kB 00:00 2024-03-29T11:10:40.213 INFO:teuthology.orchestra.run.smithi121.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.1 MB/s | 358 kB 00:00 2024-03-29T11:10:40.246 INFO:teuthology.orchestra.run.smithi121.stdout:(23/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 489 kB/s | 172 kB 00:00 2024-03-29T11:10:40.263 INFO:teuthology.orchestra.run.smithi121.stdout:(24/36): python3-jaraco-classes-3.2.1-5.el9.noa 234 kB/s | 18 kB 00:00 2024-03-29T11:10:40.288 INFO:teuthology.orchestra.run.smithi121.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 307 kB/s | 23 kB 00:00 2024-03-29T11:10:40.322 INFO:teuthology.orchestra.run.smithi121.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 258 kB/s | 19 kB 00:00 2024-03-29T11:10:40.347 INFO:teuthology.orchestra.run.smithi121.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 234 kB/s | 20 kB 00:00 2024-03-29T11:10:40.389 INFO:teuthology.orchestra.run.smithi121.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 460 kB/s | 46 kB 00:00 2024-03-29T11:10:40.415 INFO:teuthology.orchestra.run.smithi121.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 856 kB/s | 79 kB 00:00 2024-03-29T11:10:40.457 INFO:teuthology.orchestra.run.smithi121.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 245 kB/s | 16 kB 00:00 2024-03-29T11:10:40.499 INFO:teuthology.orchestra.run.smithi121.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.1 MB/s | 90 kB 00:00 2024-03-29T11:10:40.524 INFO:teuthology.orchestra.run.smithi121.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 535 kB/s | 36 kB 00:00 2024-03-29T11:10:40.549 INFO:teuthology.orchestra.run.smithi121.stdout:(33/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.3 MB/s | 272 kB 00:00 2024-03-29T11:10:40.616 INFO:teuthology.orchestra.run.smithi121.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2024-03-29T11:10:40.642 INFO:teuthology.orchestra.run.smithi121.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 217 kB/s | 20 kB 00:00 2024-03-29T11:10:40.666 INFO:teuthology.orchestra.run.smithi121.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.9 MB/s | 427 kB 00:00 2024-03-29T11:10:40.673 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:10:40.673 INFO:teuthology.orchestra.run.smithi121.stdout:Total 11 MB/s | 30 MB 00:02 2024-03-29T11:10:40.697 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-29T11:10:40.739 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-29T11:10:40.742 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-29T11:10:41.044 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-29T11:10:41.113 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:10:41.148 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-29T11:10:41.148 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-29T11:10:41.154 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:10:41.182 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:10:41.255 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:10:41.306 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:10:41.339 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:10:41.433 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:10:41.511 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:10:41.559 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:10:41.632 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:10:41.685 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:10:41.723 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:10:41.789 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:10:41.846 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:10:42.040 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:10:42.086 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:10:42.124 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:10:42.167 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:10:42.198 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:10:42.276 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:10:42.315 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:10:42.349 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:10:42.390 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:10:42.433 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:10:42.467 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:10:42.490 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-29T11:10:42.544 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:10:42.561 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:10:42.577 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:10:42.598 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:10:42.604 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:10:42.674 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:10:42.676 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:10:42.720 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:10:42.753 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:10:42.817 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:10:42.844 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:10:42.922 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:10:42.948 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:10:42.964 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:10:43.036 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:10:43.091 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:10:43.114 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:10:43.128 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:10:43.144 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:10:43.170 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:10:43.170 INFO:teuthology.orchestra.run.smithi047.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:10:43.170 INFO:teuthology.orchestra.run.smithi047.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:10:43.170 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:10:43.170 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:10:43.170 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:10:43.177 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:10:43.231 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:10:43.417 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:10:43.467 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:10:43.505 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:10:43.540 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:10:43.569 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:10:43.645 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:10:43.679 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:10:43.730 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:10:43.762 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:10:43.796 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:10:43.831 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:10:43.921 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:10:43.971 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:10:44.043 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:10:44.179 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:10:44.298 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:10:44.454 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:10:44.476 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:10:44.501 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:10:44.501 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:10:44.501 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:10:44.501 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:10:44.502 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:10:44.502 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:10:44.687 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:10:44.713 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:10:44.713 INFO:teuthology.orchestra.run.smithi047.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:10:44.713 INFO:teuthology.orchestra.run.smithi047.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:10:44.713 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:10:44.713 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:10:44.713 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:10:45.187 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:10:45.210 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:10:45.210 INFO:teuthology.orchestra.run.smithi047.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:10:45.210 INFO:teuthology.orchestra.run.smithi047.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:10:45.211 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:10:45.211 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:10:45.211 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:10:45.421 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:10:45.444 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:10:45.444 INFO:teuthology.orchestra.run.smithi047.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:10:45.444 INFO:teuthology.orchestra.run.smithi047.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:10:45.444 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:10:45.444 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:10:45.444 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:10:45.513 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:10:45.943 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:10:45.969 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:10:45.969 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:10:45.969 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:10:45.969 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:10:45.969 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:10:45.969 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:10:46.410 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:10:46.433 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:10:46.433 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:10:46.433 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:10:46.433 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:10:46.434 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:10:46.434 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:10:46.653 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:10:46.676 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:10:46.676 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:10:46.676 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:10:46.676 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:10:46.676 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:10:46.676 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:10:46.742 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:10:47.293 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:10:47.294 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:10:47.294 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:10:47.294 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:10:47.294 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:10:47.294 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:10:47.295 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:10:47.295 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:10:47.295 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:10:47.295 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:10:47.295 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:10:47.295 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:10:47.295 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:10:47.295 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:10:47.295 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:10:47.295 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:10:47.295 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:10:47.295 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:10:47.295 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:10:47.296 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:10:47.296 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:10:47.296 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:10:47.296 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:10:47.296 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:10:47.296 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:10:47.296 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:10:47.296 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:10:47.296 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:10:47.296 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:10:47.297 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:10:47.297 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:10:47.297 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:10:47.297 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:10:47.297 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:10:47.297 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:10:47.297 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:10:48.476 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:10:48.476 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:10:48.476 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-29T11:10:48.477 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:48.477 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:48.477 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:48.477 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:10:48.477 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:48.477 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:48.477 INFO:teuthology.orchestra.run.smithi047.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:10:48.477 INFO:teuthology.orchestra.run.smithi047.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:10:48.477 INFO:teuthology.orchestra.run.smithi047.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:10:48.477 INFO:teuthology.orchestra.run.smithi047.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:10:48.477 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:10:48.477 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:10:48.477 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:10:48.477 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:10:48.478 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:10:48.478 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:10:48.478 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:10:48.478 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:10:48.478 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:10:48.478 INFO:teuthology.orchestra.run.smithi047.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:10:48.478 INFO:teuthology.orchestra.run.smithi047.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:10:48.478 INFO:teuthology.orchestra.run.smithi047.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:10:48.478 INFO:teuthology.orchestra.run.smithi047.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:10:48.478 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:10:48.478 INFO:teuthology.orchestra.run.smithi047.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:10:48.478 INFO:teuthology.orchestra.run.smithi047.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:10:48.478 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:10:48.479 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:10:48.479 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:10:48.479 INFO:teuthology.orchestra.run.smithi047.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:10:48.479 INFO:teuthology.orchestra.run.smithi047.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:10:48.479 INFO:teuthology.orchestra.run.smithi047.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:10:48.479 INFO:teuthology.orchestra.run.smithi047.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:10:48.479 INFO:teuthology.orchestra.run.smithi047.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:10:48.479 INFO:teuthology.orchestra.run.smithi047.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:10:48.479 INFO:teuthology.orchestra.run.smithi047.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:10:48.479 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:10:48.479 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-29T11:10:48.483 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:10:48.483 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:10:48.483 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:10:48.483 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:10:48.483 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:10:48.483 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:10:48.483 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:10:48.483 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:10:48.484 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:10:48.484 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:10:48.484 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:10:48.484 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:10:48.486 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:10:48.486 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:10:48.486 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:10:48.486 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:10:48.487 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:10:48.487 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:10:48.487 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:10:48.487 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:10:48.487 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:10:48.487 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:10:48.487 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:10:48.487 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:10:48.487 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:10:48.487 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:10:48.487 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:10:48.487 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:10:48.487 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:10:48.487 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:10:48.488 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:10:48.488 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:10:48.488 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:10:48.488 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:10:48.488 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:10:48.488 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:10:48.844 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-base 2024-03-29T11:10:49.422 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:00 ago on Fri 29 Mar 2024 11:09:49 AM UTC. 2024-03-29T11:10:49.492 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:10:49.492 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:10:49.492 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-29T11:10:49.492 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:49.492 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:49.492 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:49.492 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:10:49.492 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:49.492 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:49.493 INFO:teuthology.orchestra.run.smithi121.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:10:49.493 INFO:teuthology.orchestra.run.smithi121.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:10:49.493 INFO:teuthology.orchestra.run.smithi121.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:10:49.493 INFO:teuthology.orchestra.run.smithi121.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:10:49.493 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:10:49.493 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:10:49.493 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:10:49.493 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:10:49.493 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:10:49.493 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:10:49.493 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:10:49.493 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:10:49.493 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:10:49.493 INFO:teuthology.orchestra.run.smithi121.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:10:49.493 INFO:teuthology.orchestra.run.smithi121.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:10:49.494 INFO:teuthology.orchestra.run.smithi121.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:10:49.494 INFO:teuthology.orchestra.run.smithi121.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:10:49.494 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:10:49.494 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:10:49.494 INFO:teuthology.orchestra.run.smithi121.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:10:49.494 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:10:49.494 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:10:49.494 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:10:49.494 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:10:49.494 INFO:teuthology.orchestra.run.smithi121.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:10:49.494 INFO:teuthology.orchestra.run.smithi121.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:10:49.495 INFO:teuthology.orchestra.run.smithi121.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:10:49.495 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:10:49.495 INFO:teuthology.orchestra.run.smithi121.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:10:49.495 INFO:teuthology.orchestra.run.smithi121.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:10:49.495 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:10:49.495 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-29T11:10:49.525 INFO:teuthology.orchestra.run.smithi047.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:10:49.580 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-29T11:10:49.581 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2024-03-29T11:10:49.582 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-29T11:10:49.641 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install cephadm 2024-03-29T11:10:49.832 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-base 2024-03-29T11:10:50.207 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:01 ago on Fri 29 Mar 2024 11:09:49 AM UTC. 2024-03-29T11:10:50.362 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-29T11:10:50.363 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-29T11:10:50.363 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2024-03-29T11:10:50.363 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-29T11:10:50.363 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-29T11:10:50.364 INFO:teuthology.orchestra.run.smithi047.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:10:50.364 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:10:50.364 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-29T11:10:50.364 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-29T11:10:50.364 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-29T11:10:50.364 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:10:50.364 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 747 k 2024-03-29T11:10:50.364 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 754 k 2024-03-29T11:10:50.364 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-29T11:10:50.396 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:53 ago on Fri 29 Mar 2024 11:09:57 AM UTC. 2024-03-29T11:10:50.495 INFO:teuthology.orchestra.run.smithi121.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:10:50.546 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-29T11:10:50.548 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-03-29T11:10:50.548 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-29T11:10:50.607 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install cephadm 2024-03-29T11:10:50.797 INFO:teuthology.orchestra.run.smithi047.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 1.7 MB/s | 747 kB 00:00 2024-03-29T11:10:50.797 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:10:50.797 INFO:teuthology.orchestra.run.smithi047.stdout:Total 1.7 MB/s | 747 kB 00:00 2024-03-29T11:10:50.798 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-29T11:10:50.803 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-29T11:10:50.803 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-29T11:10:50.809 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-29T11:10:50.809 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-29T11:10:50.887 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-29T11:10:51.171 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:54 ago on Fri 29 Mar 2024 11:09:57 AM UTC. 2024-03-29T11:10:51.314 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:10:51.324 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-29T11:10:51.324 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:10:51.324 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-03-29T11:10:51.324 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:10:51.324 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-29T11:10:51.324 INFO:teuthology.orchestra.run.smithi121.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:10:51.325 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:10:51.325 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-29T11:10:51.325 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:10:51.325 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-03-29T11:10:51.325 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:10:51.325 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 747 k 2024-03-29T11:10:51.325 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 754 k 2024-03-29T11:10:51.326 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-29T11:10:51.395 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:10:51.714 INFO:teuthology.orchestra.run.smithi121.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 1.9 MB/s | 747 kB 00:00 2024-03-29T11:10:51.714 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:10:51.714 INFO:teuthology.orchestra.run.smithi121.stdout:Total 1.9 MB/s | 747 kB 00:00 2024-03-29T11:10:51.714 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-29T11:10:51.719 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-29T11:10:51.720 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-29T11:10:51.725 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-29T11:10:51.725 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-29T11:10:51.780 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:10:51.796 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-29T11:10:52.052 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:10:52.052 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:10:52.052 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-29T11:10:52.052 INFO:teuthology.orchestra.run.smithi047.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:10:52.052 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:10:52.052 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-29T11:10:52.246 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:10:52.269 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:10:52.327 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:10:52.708 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:10:52.843 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:03 ago on Fri 29 Mar 2024 11:09:49 AM UTC. 2024-03-29T11:10:53.000 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-29T11:10:53.001 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-29T11:10:53.001 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-03-29T11:10:53.001 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-29T11:10:53.002 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-29T11:10:53.002 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:10:53.002 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:10:53.002 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-29T11:10:53.002 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-29T11:10:53.002 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-29T11:10:53.002 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:10:53.002 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 145 k 2024-03-29T11:10:53.002 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 436 k 2024-03-29T11:10:53.002 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-29T11:10:53.003 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:10:53.003 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:10:53.003 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-29T11:10:53.003 INFO:teuthology.orchestra.run.smithi121.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:10:53.003 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:10:53.003 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-29T11:10:53.201 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:10:53.361 INFO:teuthology.orchestra.run.smithi047.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 405 kB/s | 145 kB 00:00 2024-03-29T11:10:53.362 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:10:53.362 INFO:teuthology.orchestra.run.smithi047.stdout:Total 403 kB/s | 145 kB 00:00 2024-03-29T11:10:53.362 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-29T11:10:53.370 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-29T11:10:53.370 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-29T11:10:53.425 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-29T11:10:53.426 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-29T11:10:53.570 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-29T11:10:53.629 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:10:53.653 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:10:53.653 INFO:teuthology.orchestra.run.smithi047.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:10:53.653 INFO:teuthology.orchestra.run.smithi047.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:10:53.653 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:10:53.774 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:56 ago on Fri 29 Mar 2024 11:09:57 AM UTC. 2024-03-29T11:10:53.927 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-29T11:10:53.928 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:10:53.928 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-03-29T11:10:53.928 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:10:53.928 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-29T11:10:53.928 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:10:53.928 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:10:53.928 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-29T11:10:53.928 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:10:53.928 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-03-29T11:10:53.928 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:10:53.928 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 145 k 2024-03-29T11:10:53.929 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 436 k 2024-03-29T11:10:53.929 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-29T11:10:54.253 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 448 kB/s | 145 kB 00:00 2024-03-29T11:10:54.254 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:10:54.254 INFO:teuthology.orchestra.run.smithi121.stdout:Total 446 kB/s | 145 kB 00:00 2024-03-29T11:10:54.254 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-29T11:10:54.261 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-29T11:10:54.261 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-29T11:10:54.314 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-29T11:10:54.314 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-29T11:10:54.451 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-29T11:10:54.521 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:10:54.545 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:10:54.545 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:10:54.545 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:10:54.545 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:10:54.606 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:10:54.606 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:10:54.606 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-29T11:10:54.606 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:54.607 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:10:54.607 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-29T11:10:54.862 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-mgr 2024-03-29T11:10:55.450 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:06 ago on Fri 29 Mar 2024 11:09:49 AM UTC. 2024-03-29T11:10:55.516 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:10:55.516 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:10:55.516 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-29T11:10:55.516 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:10:55.516 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:10:55.516 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-29T11:10:55.551 INFO:teuthology.orchestra.run.smithi047.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:10:55.607 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-29T11:10:55.608 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2024-03-29T11:10:55.608 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-29T11:10:55.673 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:10:55.756 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr 2024-03-29T11:10:56.243 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:07 ago on Fri 29 Mar 2024 11:09:49 AM UTC. 2024-03-29T11:10:56.331 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:59 ago on Fri 29 Mar 2024 11:09:57 AM UTC. 2024-03-29T11:10:56.403 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-29T11:10:56.404 INFO:teuthology.orchestra.run.smithi047.stdout:====================================================================================== 2024-03-29T11:10:56.404 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2024-03-29T11:10:56.404 INFO:teuthology.orchestra.run.smithi047.stdout:====================================================================================== 2024-03-29T11:10:56.404 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-29T11:10:56.405 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:10:56.405 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2024-03-29T11:10:56.405 INFO:teuthology.orchestra.run.smithi047.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:10:56.405 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:10:56.405 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:10:56.405 INFO:teuthology.orchestra.run.smithi047.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:10:56.405 INFO:teuthology.orchestra.run.smithi047.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:10:56.405 INFO:teuthology.orchestra.run.smithi047.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:10:56.405 INFO:teuthology.orchestra.run.smithi047.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:10:56.405 INFO:teuthology.orchestra.run.smithi047.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:10:56.405 INFO:teuthology.orchestra.run.smithi047.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:10:56.405 INFO:teuthology.orchestra.run.smithi047.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:10:56.406 INFO:teuthology.orchestra.run.smithi047.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:10:56.406 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:10:56.406 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-29T11:10:56.406 INFO:teuthology.orchestra.run.smithi047.stdout:====================================================================================== 2024-03-29T11:10:56.406 INFO:teuthology.orchestra.run.smithi047.stdout:Install 12 Packages 2024-03-29T11:10:56.406 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:10:56.407 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 6.9 M 2024-03-29T11:10:56.407 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 26 M 2024-03-29T11:10:56.407 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-29T11:10:56.429 INFO:teuthology.orchestra.run.smithi121.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:10:56.481 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-29T11:10:56.482 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-03-29T11:10:56.482 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-29T11:10:56.544 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:10:56.734 INFO:teuthology.orchestra.run.smithi047.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 164 kB/s | 31 kB 00:00 2024-03-29T11:10:56.758 INFO:teuthology.orchestra.run.smithi047.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 70 kB/s | 15 kB 00:00 2024-03-29T11:10:56.900 INFO:teuthology.orchestra.run.smithi047.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.6 MB/s | 238 kB 00:00 2024-03-29T11:10:56.942 INFO:teuthology.orchestra.run.smithi047.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.8 MB/s | 1.0 MB 00:00 2024-03-29T11:10:57.009 INFO:teuthology.orchestra.run.smithi047.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 3.8 MB/s | 1.8 MB 00:00 2024-03-29T11:10:57.076 INFO:teuthology.orchestra.run.smithi047.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 4.8 MB/s | 863 kB 00:00 2024-03-29T11:10:57.105 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:00 ago on Fri 29 Mar 2024 11:09:57 AM UTC. 2024-03-29T11:10:57.255 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-29T11:10:57.256 INFO:teuthology.orchestra.run.smithi121.stdout:====================================================================================== 2024-03-29T11:10:57.256 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-03-29T11:10:57.256 INFO:teuthology.orchestra.run.smithi121.stdout:====================================================================================== 2024-03-29T11:10:57.256 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-29T11:10:57.256 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:10:57.256 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-03-29T11:10:57.256 INFO:teuthology.orchestra.run.smithi121.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:10:57.256 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:10:57.256 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:10:57.256 INFO:teuthology.orchestra.run.smithi121.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:10:57.256 INFO:teuthology.orchestra.run.smithi121.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:10:57.256 INFO:teuthology.orchestra.run.smithi121.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:10:57.257 INFO:teuthology.orchestra.run.smithi121.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:10:57.257 INFO:teuthology.orchestra.run.smithi121.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:10:57.257 INFO:teuthology.orchestra.run.smithi121.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:10:57.257 INFO:teuthology.orchestra.run.smithi121.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:10:57.257 INFO:teuthology.orchestra.run.smithi121.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:10:57.257 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:10:57.257 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-29T11:10:57.257 INFO:teuthology.orchestra.run.smithi121.stdout:====================================================================================== 2024-03-29T11:10:57.257 INFO:teuthology.orchestra.run.smithi121.stdout:Install 12 Packages 2024-03-29T11:10:57.257 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:10:57.258 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 6.9 M 2024-03-29T11:10:57.258 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 26 M 2024-03-29T11:10:57.258 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-29T11:10:57.259 INFO:teuthology.orchestra.run.smithi047.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 78 kB/s | 19 kB 00:00 2024-03-29T11:10:57.451 INFO:teuthology.orchestra.run.smithi047.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 753 kB/s | 144 kB 00:00 2024-03-29T11:10:57.526 INFO:teuthology.orchestra.run.smithi047.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 410 kB/s | 31 kB 00:00 2024-03-29T11:10:57.610 INFO:teuthology.orchestra.run.smithi047.stdout:(10/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 820 kB/s | 548 kB 00:00 2024-03-29T11:10:57.685 INFO:teuthology.orchestra.run.smithi047.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.2 MB/s | 188 kB 00:00 2024-03-29T11:10:57.724 INFO:teuthology.orchestra.run.smithi121.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 73 kB/s | 15 kB 00:00 2024-03-29T11:10:57.824 INFO:teuthology.orchestra.run.smithi121.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 101 kB/s | 31 kB 00:00 2024-03-29T11:10:57.883 INFO:teuthology.orchestra.run.smithi121.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.4 MB/s | 1.0 MB 00:00 2024-03-29T11:10:57.908 INFO:teuthology.orchestra.run.smithi121.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.8 MB/s | 238 kB 00:00 2024-03-29T11:10:57.967 INFO:teuthology.orchestra.run.smithi121.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 3.9 MB/s | 1.8 MB 00:00 2024-03-29T11:10:58.067 INFO:teuthology.orchestra.run.smithi121.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 195 kB/s | 19 kB 00:00 2024-03-29T11:10:58.142 INFO:teuthology.orchestra.run.smithi121.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.3 MB/s | 548 kB 00:00 2024-03-29T11:10:58.176 INFO:teuthology.orchestra.run.smithi121.stdout:(8/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.9 MB/s | 863 kB 00:00 2024-03-29T11:10:58.194 INFO:teuthology.orchestra.run.smithi047.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 1.8 MB/s | 2.0 MB 00:01 2024-03-29T11:10:58.198 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:10:58.198 INFO:teuthology.orchestra.run.smithi047.stdout:Total 3.8 MB/s | 6.9 MB 00:01 2024-03-29T11:10:58.201 INFO:teuthology.orchestra.run.smithi121.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.4 MB/s | 144 kB 00:00 2024-03-29T11:10:58.219 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-29T11:10:58.239 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-29T11:10:58.239 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-29T11:10:58.251 INFO:teuthology.orchestra.run.smithi121.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-03-29T11:10:58.276 INFO:teuthology.orchestra.run.smithi121.stdout:(11/12): python3-repoze-lru-0.7-16.el9.noarch.r 306 kB/s | 31 kB 00:00 2024-03-29T11:10:58.318 INFO:teuthology.orchestra.run.smithi121.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 8.1 MB/s | 2.0 MB 00:00 2024-03-29T11:10:58.321 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:10:58.321 INFO:teuthology.orchestra.run.smithi121.stdout:Total 6.4 MB/s | 6.9 MB 00:01 2024-03-29T11:10:58.339 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-29T11:10:58.356 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-29T11:10:58.356 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-29T11:10:58.429 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-29T11:10:58.430 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-29T11:10:58.533 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-29T11:10:58.533 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-29T11:10:59.042 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-29T11:10:59.171 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-29T11:10:59.171 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:10:59.171 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:10:59.171 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:10:59.172 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:10:59.209 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:10:59.219 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:10:59.240 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:10:59.257 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:10:59.342 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:10:59.350 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:10:59.464 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:10:59.479 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:10:59.554 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:10:59.579 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:10:59.603 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:10:59.628 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:10:59.631 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:10:59.657 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:10:59.663 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:10:59.688 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:10:59.908 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:10:59.938 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:10:59.943 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:10:59.971 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:11:00.783 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:11:00.783 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:11:00.783 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:11:00.783 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:11:00.783 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:11:00.783 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:11:00.783 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:11:00.784 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:11:00.784 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:11:00.784 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:11:00.784 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:11:00.784 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:11:00.901 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:11:00.902 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:11:00.902 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:11:00.902 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:11:00.902 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:11:00.902 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:11:00.902 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:11:00.902 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:11:00.902 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:11:00.902 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:11:00.902 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:11:00.902 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:11:01.125 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:11:01.125 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:11:01.125 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-29T11:11:01.126 INFO:teuthology.orchestra.run.smithi121.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:11:01.126 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:11:01.126 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:11:01.126 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:11:01.126 INFO:teuthology.orchestra.run.smithi121.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:11:01.126 INFO:teuthology.orchestra.run.smithi121.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:11:01.126 INFO:teuthology.orchestra.run.smithi121.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:11:01.126 INFO:teuthology.orchestra.run.smithi121.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:11:01.126 INFO:teuthology.orchestra.run.smithi121.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:11:01.126 INFO:teuthology.orchestra.run.smithi121.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:11:01.126 INFO:teuthology.orchestra.run.smithi121.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:11:01.126 INFO:teuthology.orchestra.run.smithi121.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:11:01.126 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:11:01.127 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-29T11:11:01.272 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:11:01.272 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:11:01.272 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-29T11:11:01.272 INFO:teuthology.orchestra.run.smithi047.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:11:01.273 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:11:01.273 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:11:01.273 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:11:01.273 INFO:teuthology.orchestra.run.smithi047.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:11:01.273 INFO:teuthology.orchestra.run.smithi047.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:11:01.273 INFO:teuthology.orchestra.run.smithi047.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:11:01.273 INFO:teuthology.orchestra.run.smithi047.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:11:01.273 INFO:teuthology.orchestra.run.smithi047.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:11:01.273 INFO:teuthology.orchestra.run.smithi047.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:11:01.273 INFO:teuthology.orchestra.run.smithi047.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:11:01.273 INFO:teuthology.orchestra.run.smithi047.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:11:01.273 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:11:01.273 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-29T11:11:01.375 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:11:01.653 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:11:01.930 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:04 ago on Fri 29 Mar 2024 11:09:57 AM UTC. 2024-03-29T11:11:02.085 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-29T11:11:02.086 INFO:teuthology.orchestra.run.smithi121.stdout:============================================================================================ 2024-03-29T11:11:02.086 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-03-29T11:11:02.086 INFO:teuthology.orchestra.run.smithi121.stdout:============================================================================================ 2024-03-29T11:11:02.086 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-29T11:11:02.086 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:11:02.086 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-03-29T11:11:02.086 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:11:02.087 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:11:02.087 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:11:02.087 INFO:teuthology.orchestra.run.smithi121.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:11:02.087 INFO:teuthology.orchestra.run.smithi121.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:11:02.087 INFO:teuthology.orchestra.run.smithi121.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:11:02.087 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:11:02.087 INFO:teuthology.orchestra.run.smithi121.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:11:02.087 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:11:02.087 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:11:02.087 INFO:teuthology.orchestra.run.smithi121.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:11:02.087 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:11:02.087 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-29T11:11:02.087 INFO:teuthology.orchestra.run.smithi121.stdout:============================================================================================ 2024-03-29T11:11:02.087 INFO:teuthology.orchestra.run.smithi121.stdout:Install 12 Packages 2024-03-29T11:11:02.088 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:11:02.088 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 37 M 2024-03-29T11:11:02.088 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 209 M 2024-03-29T11:11:02.088 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-29T11:11:02.226 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:13 ago on Fri 29 Mar 2024 11:09:49 AM UTC. 2024-03-29T11:11:02.384 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-29T11:11:02.385 INFO:teuthology.orchestra.run.smithi047.stdout:============================================================================================ 2024-03-29T11:11:02.385 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2024-03-29T11:11:02.385 INFO:teuthology.orchestra.run.smithi047.stdout:============================================================================================ 2024-03-29T11:11:02.385 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-29T11:11:02.385 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:11:02.386 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2024-03-29T11:11:02.386 INFO:teuthology.orchestra.run.smithi047.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:11:02.386 INFO:teuthology.orchestra.run.smithi047.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:11:02.386 INFO:teuthology.orchestra.run.smithi047.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:11:02.386 INFO:teuthology.orchestra.run.smithi047.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:11:02.386 INFO:teuthology.orchestra.run.smithi047.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:11:02.386 INFO:teuthology.orchestra.run.smithi047.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:11:02.386 INFO:teuthology.orchestra.run.smithi047.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:11:02.386 INFO:teuthology.orchestra.run.smithi047.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:11:02.386 INFO:teuthology.orchestra.run.smithi047.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:11:02.386 INFO:teuthology.orchestra.run.smithi047.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:11:02.386 INFO:teuthology.orchestra.run.smithi047.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:11:02.386 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:11:02.386 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-29T11:11:02.387 INFO:teuthology.orchestra.run.smithi047.stdout:============================================================================================ 2024-03-29T11:11:02.387 INFO:teuthology.orchestra.run.smithi047.stdout:Install 12 Packages 2024-03-29T11:11:02.387 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:11:02.387 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 37 M 2024-03-29T11:11:02.387 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 209 M 2024-03-29T11:11:02.387 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-29T11:11:02.442 INFO:teuthology.orchestra.run.smithi121.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 207 kB/s | 32 kB 00:00 2024-03-29T11:11:02.475 INFO:teuthology.orchestra.run.smithi121.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 537 kB/s | 17 kB 00:00 2024-03-29T11:11:02.517 INFO:teuthology.orchestra.run.smithi121.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 847 kB/s | 34 kB 00:00 2024-03-29T11:11:02.576 INFO:teuthology.orchestra.run.smithi121.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 10 MB/s | 3.0 MB 00:00 2024-03-29T11:11:02.776 INFO:teuthology.orchestra.run.smithi121.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 18 MB/s | 4.7 MB 00:00 2024-03-29T11:11:02.809 INFO:teuthology.orchestra.run.smithi047.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 195 kB/s | 32 kB 00:00 2024-03-29T11:11:02.842 INFO:teuthology.orchestra.run.smithi047.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 529 kB/s | 17 kB 00:00 2024-03-29T11:11:02.843 INFO:teuthology.orchestra.run.smithi121.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 18 MB/s | 4.8 MB 00:00 2024-03-29T11:11:02.868 INFO:teuthology.orchestra.run.smithi121.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-29T11:11:02.884 INFO:teuthology.orchestra.run.smithi047.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 819 kB/s | 34 kB 00:00 2024-03-29T11:11:03.043 INFO:teuthology.orchestra.run.smithi121.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.5 MB/s | 803 kB 00:00 2024-03-29T11:11:03.094 INFO:teuthology.orchestra.run.smithi121.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 4.0 MB/s | 195 kB 00:00 2024-03-29T11:11:03.253 INFO:teuthology.orchestra.run.smithi121.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 38 MB/s | 16 MB 00:00 2024-03-29T11:11:03.302 INFO:teuthology.orchestra.run.smithi047.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 11 MB/s | 4.7 MB 00:00 2024-03-29T11:11:03.428 INFO:teuthology.orchestra.run.smithi121.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 740 kB/s | 245 kB 00:00 2024-03-29T11:11:03.460 INFO:teuthology.orchestra.run.smithi047.stdout:(5/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.7 MB/s | 3.0 MB 00:00 2024-03-29T11:11:03.535 INFO:teuthology.orchestra.run.smithi047.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 21 MB/s | 4.8 MB 00:00 2024-03-29T11:11:03.561 INFO:teuthology.orchestra.run.smithi047.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2024-03-29T11:11:03.745 INFO:teuthology.orchestra.run.smithi047.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.3 MB/s | 803 kB 00:00 2024-03-29T11:11:03.787 INFO:teuthology.orchestra.run.smithi047.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 4.9 MB/s | 195 kB 00:00 2024-03-29T11:11:03.836 INFO:teuthology.orchestra.run.smithi121.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 4.8 MB/s | 7.4 MB 00:01 2024-03-29T11:11:03.838 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:11:03.839 INFO:teuthology.orchestra.run.smithi121.stdout:Total 21 MB/s | 37 MB 00:01 2024-03-29T11:11:03.877 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-29T11:11:03.901 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-29T11:11:03.901 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-29T11:11:03.936 INFO:teuthology.orchestra.run.smithi047.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 39 MB/s | 16 MB 00:00 2024-03-29T11:11:03.962 INFO:teuthology.orchestra.run.smithi047.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.4 MB/s | 245 kB 00:00 2024-03-29T11:11:04.252 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-29T11:11:04.253 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-29T11:11:04.570 INFO:teuthology.orchestra.run.smithi047.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 3.8 MB/s | 7.4 MB 00:01 2024-03-29T11:11:04.573 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:11:04.573 INFO:teuthology.orchestra.run.smithi047.stdout:Total 17 MB/s | 37 MB 00:02 2024-03-29T11:11:04.614 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-29T11:11:04.639 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-29T11:11:04.640 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-29T11:11:04.960 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-29T11:11:05.018 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-29T11:11:05.019 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-29T11:11:05.025 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:11:05.065 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:11:05.158 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:11:05.202 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:11:05.378 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:11:05.409 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:11:05.485 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:11:05.740 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-29T11:11:05.819 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:11:05.849 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:11:05.947 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:11:05.994 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:11:06.128 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:11:06.180 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:11:06.210 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:11:06.246 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:11:06.295 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:11:06.987 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:11:07.101 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:11:07.463 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:11:08.410 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:11:08.482 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:11:08.510 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:11:09.499 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:11:09.526 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:11:11.050 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:11:11.051 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:11:11.051 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:11:11.051 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:11:11.051 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:11:11.051 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:11:11.051 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:11:11.051 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:11:11.051 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:11:11.051 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:11:11.051 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:11:11.051 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:11:11.483 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:11:11.483 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:11:11.483 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-29T11:11:11.483 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:11:11.483 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:11:11.484 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:11:11.484 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:11:11.484 INFO:teuthology.orchestra.run.smithi121.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:11:11.484 INFO:teuthology.orchestra.run.smithi121.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:11:11.484 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:11:11.484 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:11:11.484 INFO:teuthology.orchestra.run.smithi121.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:11:11.484 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:11:11.484 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:11:11.484 INFO:teuthology.orchestra.run.smithi121.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:11:11.484 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:11:11.484 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-29T11:11:11.835 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:11:12.010 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:11:12.010 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:11:12.011 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:11:12.011 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:11:12.011 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:11:12.011 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:11:12.011 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:11:12.011 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:11:12.011 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:11:12.011 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:11:12.011 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:11:12.011 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:11:12.381 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:11:12.382 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:11:12.382 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-29T11:11:12.382 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:11:12.382 INFO:teuthology.orchestra.run.smithi047.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:11:12.382 INFO:teuthology.orchestra.run.smithi047.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:11:12.382 INFO:teuthology.orchestra.run.smithi047.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:11:12.382 INFO:teuthology.orchestra.run.smithi047.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:11:12.383 INFO:teuthology.orchestra.run.smithi047.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:11:12.383 INFO:teuthology.orchestra.run.smithi047.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:11:12.383 INFO:teuthology.orchestra.run.smithi047.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:11:12.383 INFO:teuthology.orchestra.run.smithi047.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:11:12.383 INFO:teuthology.orchestra.run.smithi047.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:11:12.383 INFO:teuthology.orchestra.run.smithi047.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:11:12.383 INFO:teuthology.orchestra.run.smithi047.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:11:12.383 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:11:12.383 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-29T11:11:12.404 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:15 ago on Fri 29 Mar 2024 11:09:57 AM UTC. 2024-03-29T11:11:12.558 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-29T11:11:12.559 INFO:teuthology.orchestra.run.smithi121.stdout:======================================================================================== 2024-03-29T11:11:12.559 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-03-29T11:11:12.559 INFO:teuthology.orchestra.run.smithi121.stdout:======================================================================================== 2024-03-29T11:11:12.559 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-29T11:11:12.559 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:11:12.559 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-03-29T11:11:12.559 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:11:12.559 INFO:teuthology.orchestra.run.smithi121.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:11:12.559 INFO:teuthology.orchestra.run.smithi121.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:11:12.559 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:11:12.559 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:11:12.559 INFO:teuthology.orchestra.run.smithi121.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:11:12.560 INFO:teuthology.orchestra.run.smithi121.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:11:12.560 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:11:12.560 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:11:12.560 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:11:12.560 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:11:12.560 INFO:teuthology.orchestra.run.smithi121.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:11:12.560 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:11:12.560 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-29T11:11:12.560 INFO:teuthology.orchestra.run.smithi121.stdout:======================================================================================== 2024-03-29T11:11:12.560 INFO:teuthology.orchestra.run.smithi121.stdout:Install 13 Packages 2024-03-29T11:11:12.560 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:11:12.561 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 2.0 M 2024-03-29T11:11:12.561 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 26 M 2024-03-29T11:11:12.561 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-29T11:11:12.745 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:11:12.761 INFO:teuthology.orchestra.run.smithi121.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 200 kB/s | 24 kB 00:00 2024-03-29T11:11:12.777 INFO:teuthology.orchestra.run.smithi121.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 130 kB/s | 18 kB 00:00 2024-03-29T11:11:12.828 INFO:teuthology.orchestra.run.smithi121.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-03-29T11:11:12.853 INFO:teuthology.orchestra.run.smithi121.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-29T11:11:12.878 INFO:teuthology.orchestra.run.smithi121.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T11:11:12.904 INFO:teuthology.orchestra.run.smithi121.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 859 kB/s | 43 kB 00:00 2024-03-29T11:11:12.929 INFO:teuthology.orchestra.run.smithi121.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 180 kB/s | 52 kB 00:00 2024-03-29T11:11:13.279 INFO:teuthology.orchestra.run.smithi121.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 40 kB/s | 15 kB 00:00 2024-03-29T11:11:13.321 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:24 ago on Fri 29 Mar 2024 11:09:49 AM UTC. 2024-03-29T11:11:13.338 INFO:teuthology.orchestra.run.smithi121.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 70 kB/s | 32 kB 00:00 2024-03-29T11:11:13.477 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-29T11:11:13.478 INFO:teuthology.orchestra.run.smithi047.stdout:======================================================================================== 2024-03-29T11:11:13.478 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2024-03-29T11:11:13.478 INFO:teuthology.orchestra.run.smithi047.stdout:======================================================================================== 2024-03-29T11:11:13.479 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-29T11:11:13.479 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:11:13.479 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2024-03-29T11:11:13.479 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:11:13.479 INFO:teuthology.orchestra.run.smithi047.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:11:13.479 INFO:teuthology.orchestra.run.smithi047.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:11:13.479 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:11:13.479 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:11:13.479 INFO:teuthology.orchestra.run.smithi047.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:11:13.479 INFO:teuthology.orchestra.run.smithi047.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:11:13.479 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:11:13.479 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:11:13.479 INFO:teuthology.orchestra.run.smithi047.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:11:13.479 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:11:13.480 INFO:teuthology.orchestra.run.smithi047.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:11:13.480 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:11:13.480 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-29T11:11:13.480 INFO:teuthology.orchestra.run.smithi047.stdout:======================================================================================== 2024-03-29T11:11:13.480 INFO:teuthology.orchestra.run.smithi047.stdout:Install 13 Packages 2024-03-29T11:11:13.480 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:11:13.480 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 2.0 M 2024-03-29T11:11:13.481 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 26 M 2024-03-29T11:11:13.481 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-29T11:11:13.546 INFO:teuthology.orchestra.run.smithi121.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 284 kB/s | 59 kB 00:00 2024-03-29T11:11:13.580 INFO:teuthology.orchestra.run.smithi121.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 311 kB/s | 203 kB 00:00 2024-03-29T11:11:13.660 INFO:teuthology.orchestra.run.smithi047.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 190 kB/s | 24 kB 00:00 2024-03-29T11:11:13.677 INFO:teuthology.orchestra.run.smithi047.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 124 kB/s | 18 kB 00:00 2024-03-29T11:11:13.727 INFO:teuthology.orchestra.run.smithi047.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-03-29T11:11:13.752 INFO:teuthology.orchestra.run.smithi047.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-29T11:11:13.755 INFO:teuthology.orchestra.run.smithi121.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 430 kB/s | 90 kB 00:00 2024-03-29T11:11:13.777 INFO:teuthology.orchestra.run.smithi047.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T11:11:13.803 INFO:teuthology.orchestra.run.smithi047.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 855 kB/s | 43 kB 00:00 2024-03-29T11:11:13.828 INFO:teuthology.orchestra.run.smithi047.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 176 kB/s | 52 kB 00:00 2024-03-29T11:11:13.864 INFO:teuthology.orchestra.run.smithi121.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.7 MB/s | 1.0 MB 00:00 2024-03-29T11:11:13.868 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:11:13.868 INFO:teuthology.orchestra.run.smithi121.stdout:Total 1.6 MB/s | 2.0 MB 00:01 2024-03-29T11:11:13.869 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-29T11:11:13.883 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-29T11:11:13.883 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-29T11:11:13.976 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-29T11:11:13.976 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-29T11:11:14.486 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-29T11:11:14.662 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:11:14.723 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:11:14.778 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:11:14.816 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:11:14.857 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:11:14.920 INFO:teuthology.orchestra.run.smithi047.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 28 kB/s | 32 kB 00:01 2024-03-29T11:11:14.960 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:11:15.076 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:11:15.112 INFO:teuthology.orchestra.run.smithi047.stdout:(9/13): python3-kubernetes-26.1.0-2.el9.noarch. 5.3 MB/s | 1.0 MB 00:00 2024-03-29T11:11:15.134 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:11:15.171 INFO:teuthology.orchestra.run.smithi047.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-03-29T11:11:15.237 INFO:teuthology.orchestra.run.smithi047.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 1.3 MB/s | 90 kB 00:00 2024-03-29T11:11:15.867 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:11:15.914 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:11:15.948 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:11:15.996 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:11:16.018 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:11:16.781 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:11:16.781 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:11:16.781 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:11:16.781 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:11:16.781 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:11:16.781 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:11:16.781 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:11:16.781 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:11:16.781 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:11:16.781 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:11:16.781 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:11:16.782 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:11:16.782 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:11:17.112 INFO:teuthology.orchestra.run.smithi047.stdout:(12/13): python3-google-auth-2.28.2-1.el9.noarc 62 kB/s | 203 kB 00:03 2024-03-29T11:11:17.140 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:11:17.141 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:11:17.141 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-29T11:11:17.141 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:11:17.141 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:11:17.141 INFO:teuthology.orchestra.run.smithi121.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:11:17.141 INFO:teuthology.orchestra.run.smithi121.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:11:17.141 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:11:17.141 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:11:17.141 INFO:teuthology.orchestra.run.smithi121.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:11:17.141 INFO:teuthology.orchestra.run.smithi121.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:11:17.141 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:11:17.141 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:11:17.141 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:11:17.142 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:11:17.142 INFO:teuthology.orchestra.run.smithi121.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:11:17.142 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:11:17.142 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-29T11:11:17.398 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:11:17.963 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:20 ago on Fri 29 Mar 2024 11:09:57 AM UTC. 2024-03-29T11:11:18.115 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-29T11:11:18.116 INFO:teuthology.orchestra.run.smithi121.stdout:======================================================================================== 2024-03-29T11:11:18.116 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-03-29T11:11:18.116 INFO:teuthology.orchestra.run.smithi121.stdout:======================================================================================== 2024-03-29T11:11:18.116 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-29T11:11:18.116 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:11:18.116 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-03-29T11:11:18.116 INFO:teuthology.orchestra.run.smithi121.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:11:18.117 INFO:teuthology.orchestra.run.smithi121.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:11:18.117 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:11:18.117 INFO:teuthology.orchestra.run.smithi121.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:11:18.117 INFO:teuthology.orchestra.run.smithi121.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:11:18.117 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:11:18.117 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-29T11:11:18.117 INFO:teuthology.orchestra.run.smithi121.stdout:======================================================================================== 2024-03-29T11:11:18.117 INFO:teuthology.orchestra.run.smithi121.stdout:Install 6 Packages 2024-03-29T11:11:18.117 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:11:18.118 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 6.8 M 2024-03-29T11:11:18.118 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 33 M 2024-03-29T11:11:18.118 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-29T11:11:18.732 INFO:teuthology.orchestra.run.smithi121.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 5.8 MB 00:00 2024-03-29T11:11:18.874 INFO:teuthology.orchestra.run.smithi121.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 314 kB/s | 138 kB 00:00 2024-03-29T11:11:19.041 INFO:teuthology.orchestra.run.smithi121.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 346 kB/s | 58 kB 00:00 2024-03-29T11:11:19.058 INFO:teuthology.orchestra.run.smithi121.stdout:(4/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 398 kB/s | 248 kB 00:00 2024-03-29T11:11:19.100 INFO:teuthology.orchestra.run.smithi121.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 870 kB/s | 51 kB 00:00 2024-03-29T11:11:29.070 INFO:teuthology.orchestra.run.smithi047.stdout:(13/13): python3-certifi-2021.10.8-2.el9.noarch 1.0 kB/s | 15 kB 00:15 2024-03-29T11:11:29.072 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:11:29.072 INFO:teuthology.orchestra.run.smithi047.stdout:Total 135 kB/s | 2.0 MB 00:15 2024-03-29T11:11:29.073 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-29T11:11:29.089 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-29T11:11:29.089 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-29T11:11:29.191 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-29T11:11:29.191 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-29T11:11:29.713 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-29T11:11:29.890 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:11:29.948 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:11:30.006 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:11:30.041 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:11:30.083 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:11:30.180 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:11:30.290 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:11:30.352 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:11:31.113 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:11:31.173 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:11:31.206 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:11:31.255 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:11:31.284 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:11:32.077 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:11:32.077 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:11:32.077 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:11:32.077 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:11:32.077 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:11:32.077 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:11:32.077 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:11:32.077 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:11:32.077 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:11:32.078 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:11:32.078 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:11:32.078 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:11:32.078 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:11:32.435 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:11:32.436 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:11:32.436 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-29T11:11:32.436 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:11:32.436 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:11:32.436 INFO:teuthology.orchestra.run.smithi047.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:11:32.436 INFO:teuthology.orchestra.run.smithi047.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:11:32.436 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:11:32.436 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:11:32.436 INFO:teuthology.orchestra.run.smithi047.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:11:32.436 INFO:teuthology.orchestra.run.smithi047.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:11:32.436 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:11:32.436 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:11:32.436 INFO:teuthology.orchestra.run.smithi047.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:11:32.437 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:11:32.437 INFO:teuthology.orchestra.run.smithi047.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:11:32.437 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:11:32.437 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-29T11:11:32.677 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:11:33.259 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:44 ago on Fri 29 Mar 2024 11:09:49 AM UTC. 2024-03-29T11:11:33.420 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-29T11:11:33.421 INFO:teuthology.orchestra.run.smithi047.stdout:======================================================================================== 2024-03-29T11:11:33.421 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2024-03-29T11:11:33.421 INFO:teuthology.orchestra.run.smithi047.stdout:======================================================================================== 2024-03-29T11:11:33.421 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-29T11:11:33.421 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:11:33.421 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2024-03-29T11:11:33.421 INFO:teuthology.orchestra.run.smithi047.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:11:33.421 INFO:teuthology.orchestra.run.smithi047.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:11:33.421 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:11:33.422 INFO:teuthology.orchestra.run.smithi047.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:11:33.422 INFO:teuthology.orchestra.run.smithi047.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:11:33.422 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:11:33.422 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-29T11:11:33.422 INFO:teuthology.orchestra.run.smithi047.stdout:======================================================================================== 2024-03-29T11:11:33.422 INFO:teuthology.orchestra.run.smithi047.stdout:Install 6 Packages 2024-03-29T11:11:33.422 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:11:33.423 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 6.8 M 2024-03-29T11:11:33.423 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 33 M 2024-03-29T11:11:33.423 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-29T11:11:34.048 INFO:teuthology.orchestra.run.smithi047.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 23 MB/s | 5.8 MB 00:00 2024-03-29T11:11:34.074 INFO:teuthology.orchestra.run.smithi047.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 882 kB/s | 248 kB 00:00 2024-03-29T11:11:34.141 INFO:teuthology.orchestra.run.smithi047.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 396 kB/s | 138 kB 00:00 2024-03-29T11:11:34.441 INFO:teuthology.orchestra.run.smithi047.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 157 kB/s | 58 kB 00:00 2024-03-29T11:11:34.474 INFO:teuthology.orchestra.run.smithi047.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 179 kB/s | 51 kB 00:00 2024-03-29T11:11:34.558 INFO:teuthology.orchestra.run.smithi047.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.1 MB/s | 548 kB 00:00 2024-03-29T11:11:34.561 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:11:34.561 INFO:teuthology.orchestra.run.smithi047.stdout:Total 6.0 MB/s | 6.8 MB 00:01 2024-03-29T11:11:34.574 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-29T11:11:34.585 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-29T11:11:34.585 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-29T11:11:34.677 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-29T11:11:34.677 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-29T11:11:34.831 INFO:teuthology.orchestra.run.smithi121.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 34 kB/s | 548 kB 00:16 2024-03-29T11:11:34.832 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:11:34.832 INFO:teuthology.orchestra.run.smithi121.stdout:Total 416 kB/s | 6.8 MB 00:16 2024-03-29T11:11:34.848 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-29T11:11:34.859 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-29T11:11:34.859 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-29T11:11:34.952 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-29T11:11:34.952 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-29T11:11:34.985 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-29T11:11:35.164 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:11:35.219 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:11:35.246 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-29T11:11:35.406 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:11:35.457 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:11:35.768 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:11:35.856 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:11:35.926 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:11:35.955 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:11:35.972 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:11:36.068 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:11:36.130 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:11:36.153 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:11:36.718 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:11:36.718 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:11:36.718 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:11:36.718 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:11:36.718 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:11:36.718 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:11:36.958 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:11:36.958 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:11:36.958 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:11:36.958 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:11:36.958 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:11:36.958 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:11:37.023 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:11:37.023 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:11:37.023 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-29T11:11:37.024 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:11:37.024 INFO:teuthology.orchestra.run.smithi047.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:11:37.024 INFO:teuthology.orchestra.run.smithi047.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:11:37.024 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:11:37.024 INFO:teuthology.orchestra.run.smithi047.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:11:37.024 INFO:teuthology.orchestra.run.smithi047.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:11:37.024 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:11:37.024 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-29T11:11:37.247 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-fuse 2024-03-29T11:11:37.253 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:11:37.253 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:11:37.253 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-29T11:11:37.253 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:11:37.253 INFO:teuthology.orchestra.run.smithi121.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:11:37.253 INFO:teuthology.orchestra.run.smithi121.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:11:37.253 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:11:37.254 INFO:teuthology.orchestra.run.smithi121.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:11:37.254 INFO:teuthology.orchestra.run.smithi121.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:11:37.254 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:11:37.254 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-29T11:11:37.481 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-fuse 2024-03-29T11:11:37.819 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:48 ago on Fri 29 Mar 2024 11:09:49 AM UTC. 2024-03-29T11:11:37.977 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-29T11:11:37.978 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-29T11:11:37.978 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2024-03-29T11:11:37.978 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-29T11:11:37.978 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-29T11:11:37.978 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:11:37.978 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2024-03-29T11:11:37.978 INFO:teuthology.orchestra.run.smithi047.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:11:37.978 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:11:37.978 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-29T11:11:37.978 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-29T11:11:37.978 INFO:teuthology.orchestra.run.smithi047.stdout:Install 2 Packages 2024-03-29T11:11:37.978 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:11:37.979 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 964 k 2024-03-29T11:11:37.979 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 2.8 M 2024-03-29T11:11:37.979 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-29T11:11:38.041 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Mar 2024 11:09:57 AM UTC. 2024-03-29T11:11:38.109 INFO:teuthology.orchestra.run.smithi047.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 602 kB/s | 79 kB 00:00 2024-03-29T11:11:38.191 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-29T11:11:38.192 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:11:38.192 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-03-29T11:11:38.192 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:11:38.192 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-29T11:11:38.192 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:11:38.192 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-03-29T11:11:38.192 INFO:teuthology.orchestra.run.smithi121.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:11:38.192 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:11:38.193 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-29T11:11:38.193 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:11:38.193 INFO:teuthology.orchestra.run.smithi121.stdout:Install 2 Packages 2024-03-29T11:11:38.193 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:11:38.193 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 964 k 2024-03-29T11:11:38.193 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 2.8 M 2024-03-29T11:11:38.193 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-29T11:11:38.324 INFO:teuthology.orchestra.run.smithi121.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 598 kB/s | 79 kB 00:00 2024-03-29T11:11:38.468 INFO:teuthology.orchestra.run.smithi047.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 1.8 MB/s | 885 kB 00:00 2024-03-29T11:11:38.469 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:11:38.469 INFO:teuthology.orchestra.run.smithi047.stdout:Total 1.9 MB/s | 964 kB 00:00 2024-03-29T11:11:38.469 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-29T11:11:38.491 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-29T11:11:38.491 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-29T11:11:38.567 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-29T11:11:38.567 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-29T11:11:38.699 INFO:teuthology.orchestra.run.smithi121.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 1.7 MB/s | 885 kB 00:00 2024-03-29T11:11:38.700 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:11:38.701 INFO:teuthology.orchestra.run.smithi121.stdout:Total 1.8 MB/s | 964 kB 00:00 2024-03-29T11:11:38.701 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-29T11:11:38.722 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-29T11:11:38.722 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-29T11:11:38.750 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-29T11:11:38.796 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-29T11:11:38.796 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-29T11:11:38.906 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:11:38.976 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-29T11:11:38.987 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:11:39.117 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:11:39.183 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:11:39.658 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:11:39.658 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:11:39.808 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:11:39.809 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:11:39.983 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:11:39.983 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:11:39.983 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-29T11:11:39.983 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:11:39.983 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:11:39.983 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-29T11:11:40.109 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:11:40.109 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:11:40.110 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-29T11:11:40.110 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:11:40.110 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:11:40.110 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-29T11:11:40.264 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-volume 2024-03-29T11:11:40.412 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-volume 2024-03-29T11:11:40.846 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:51 ago on Fri 29 Mar 2024 11:09:49 AM UTC. 2024-03-29T11:11:40.987 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:44 ago on Fri 29 Mar 2024 11:09:57 AM UTC. 2024-03-29T11:11:41.007 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-29T11:11:41.009 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-29T11:11:41.009 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2024-03-29T11:11:41.009 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-29T11:11:41.009 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-29T11:11:41.009 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:11:41.010 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2024-03-29T11:11:41.010 INFO:teuthology.orchestra.run.smithi047.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:11:41.010 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:11:41.010 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-29T11:11:41.010 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-29T11:11:41.010 INFO:teuthology.orchestra.run.smithi047.stdout:Install 2 Packages 2024-03-29T11:11:41.011 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:11:41.011 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 332 k 2024-03-29T11:11:41.011 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 1.5 M 2024-03-29T11:11:41.011 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-29T11:11:41.139 INFO:teuthology.orchestra.run.smithi047.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 528 kB/s | 69 kB 00:00 2024-03-29T11:11:41.142 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-29T11:11:41.142 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:11:41.142 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-03-29T11:11:41.143 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:11:41.143 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-29T11:11:41.143 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:11:41.143 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-03-29T11:11:41.143 INFO:teuthology.orchestra.run.smithi121.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:11:41.143 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:11:41.144 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-29T11:11:41.144 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:11:41.144 INFO:teuthology.orchestra.run.smithi121.stdout:Install 2 Packages 2024-03-29T11:11:41.144 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:11:41.145 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 332 k 2024-03-29T11:11:41.145 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 1.5 M 2024-03-29T11:11:41.145 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-29T11:11:41.354 INFO:teuthology.orchestra.run.smithi121.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 326 kB/s | 69 kB 00:00 2024-03-29T11:11:41.396 INFO:teuthology.orchestra.run.smithi121.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 1.0 MB/s | 264 kB 00:00 2024-03-29T11:11:41.397 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:11:41.398 INFO:teuthology.orchestra.run.smithi121.stdout:Total 1.3 MB/s | 332 kB 00:00 2024-03-29T11:11:41.398 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-29T11:11:41.407 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-29T11:11:41.407 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-29T11:11:41.441 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-29T11:11:41.441 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-29T11:11:41.481 INFO:teuthology.orchestra.run.smithi047.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 559 kB/s | 264 kB 00:00 2024-03-29T11:11:41.483 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:11:41.483 INFO:teuthology.orchestra.run.smithi047.stdout:Total 700 kB/s | 332 kB 00:00 2024-03-29T11:11:41.483 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-29T11:11:41.494 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-29T11:11:41.494 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-29T11:11:41.535 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-29T11:11:41.535 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-29T11:11:41.585 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-29T11:11:41.684 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-29T11:11:41.764 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:11:41.801 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:11:41.817 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:11:41.817 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:11:41.817 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:11:41.817 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:11:41.863 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:11:41.901 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:11:41.923 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:11:41.923 INFO:teuthology.orchestra.run.smithi047.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:11:41.924 INFO:teuthology.orchestra.run.smithi047.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:11:41.924 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:11:42.576 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:11:42.694 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:11:42.911 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:11:42.911 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:11:42.911 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-29T11:11:42.911 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:11:42.911 INFO:teuthology.orchestra.run.smithi121.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:11:42.912 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:11:42.912 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-29T11:11:43.034 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:11:43.034 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:11:43.034 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-29T11:11:43.034 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:11:43.034 INFO:teuthology.orchestra.run.smithi047.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:11:43.034 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:11:43.034 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-29T11:11:43.216 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install librados-devel 2024-03-29T11:11:43.313 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install librados-devel 2024-03-29T11:11:43.792 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:46 ago on Fri 29 Mar 2024 11:09:57 AM UTC. 2024-03-29T11:11:43.894 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:54 ago on Fri 29 Mar 2024 11:09:49 AM UTC. 2024-03-29T11:11:43.941 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-29T11:11:43.941 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:11:43.941 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-03-29T11:11:43.941 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:11:43.941 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-29T11:11:43.941 INFO:teuthology.orchestra.run.smithi121.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:11:43.941 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:11:43.942 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-29T11:11:43.942 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:11:43.942 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-03-29T11:11:43.942 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:11:43.942 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 127 k 2024-03-29T11:11:43.942 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 456 k 2024-03-29T11:11:43.942 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-29T11:11:44.051 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-29T11:11:44.052 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-29T11:11:44.052 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-03-29T11:11:44.052 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-29T11:11:44.052 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-29T11:11:44.052 INFO:teuthology.orchestra.run.smithi047.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:11:44.052 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:11:44.052 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-29T11:11:44.052 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-29T11:11:44.052 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-29T11:11:44.053 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:11:44.053 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 127 k 2024-03-29T11:11:44.053 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 456 k 2024-03-29T11:11:44.053 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-29T11:11:44.279 INFO:teuthology.orchestra.run.smithi121.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 376 kB/s | 127 kB 00:00 2024-03-29T11:11:44.280 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:11:44.280 INFO:teuthology.orchestra.run.smithi121.stdout:Total 375 kB/s | 127 kB 00:00 2024-03-29T11:11:44.280 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-29T11:11:44.285 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-29T11:11:44.285 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-29T11:11:44.341 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-29T11:11:44.341 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-29T11:11:44.371 INFO:teuthology.orchestra.run.smithi047.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 398 kB/s | 127 kB 00:00 2024-03-29T11:11:44.372 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:11:44.372 INFO:teuthology.orchestra.run.smithi047.stdout:Total 396 kB/s | 127 kB 00:00 2024-03-29T11:11:44.372 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-29T11:11:44.378 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-29T11:11:44.378 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-29T11:11:44.437 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-29T11:11:44.437 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-29T11:11:44.487 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-29T11:11:44.596 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-29T11:11:44.602 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:11:44.690 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:11:45.062 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:11:45.141 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:11:45.339 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:11:45.340 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:11:45.340 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-29T11:11:45.340 INFO:teuthology.orchestra.run.smithi121.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:11:45.340 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:11:45.340 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-29T11:11:45.425 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:11:45.425 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:11:45.425 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-29T11:11:45.426 INFO:teuthology.orchestra.run.smithi047.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:11:45.426 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:11:45.426 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-29T11:11:45.564 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install libcephfs2 2024-03-29T11:11:45.640 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install libcephfs2 2024-03-29T11:11:46.130 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:49 ago on Fri 29 Mar 2024 11:09:57 AM UTC. 2024-03-29T11:11:46.218 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:57 ago on Fri 29 Mar 2024 11:09:49 AM UTC. 2024-03-29T11:11:46.232 INFO:teuthology.orchestra.run.smithi121.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:11:46.286 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-29T11:11:46.286 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-03-29T11:11:46.286 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-29T11:11:46.320 INFO:teuthology.orchestra.run.smithi047.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:11:46.342 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install libcephfs-devel 2024-03-29T11:11:46.377 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-29T11:11:46.378 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2024-03-29T11:11:46.378 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-29T11:11:46.447 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install libcephfs-devel 2024-03-29T11:11:46.905 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:49 ago on Fri 29 Mar 2024 11:09:57 AM UTC. 2024-03-29T11:11:47.025 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:58 ago on Fri 29 Mar 2024 11:09:49 AM UTC. 2024-03-29T11:11:47.057 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-29T11:11:47.058 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:11:47.058 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-03-29T11:11:47.058 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:11:47.058 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-29T11:11:47.059 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:11:47.059 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:11:47.059 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-29T11:11:47.059 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:11:47.059 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-03-29T11:11:47.059 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:11:47.059 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 31 k 2024-03-29T11:11:47.059 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 140 k 2024-03-29T11:11:47.059 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-29T11:11:47.185 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-29T11:11:47.186 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-29T11:11:47.186 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-03-29T11:11:47.186 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-29T11:11:47.186 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-29T11:11:47.186 INFO:teuthology.orchestra.run.smithi047.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:11:47.186 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:11:47.186 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-29T11:11:47.186 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-29T11:11:47.187 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-29T11:11:47.187 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:11:47.187 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 31 k 2024-03-29T11:11:47.187 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 140 k 2024-03-29T11:11:47.187 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-29T11:11:47.316 INFO:teuthology.orchestra.run.smithi121.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 121 kB/s | 31 kB 00:00 2024-03-29T11:11:47.317 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:11:47.317 INFO:teuthology.orchestra.run.smithi121.stdout:Total 121 kB/s | 31 kB 00:00 2024-03-29T11:11:47.317 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-29T11:11:47.321 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-29T11:11:47.321 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-29T11:11:47.351 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-29T11:11:47.351 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-29T11:11:47.358 INFO:teuthology.orchestra.run.smithi047.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 182 kB/s | 31 kB 00:00 2024-03-29T11:11:47.359 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:11:47.359 INFO:teuthology.orchestra.run.smithi047.stdout:Total 181 kB/s | 31 kB 00:00 2024-03-29T11:11:47.359 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-29T11:11:47.362 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-29T11:11:47.363 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-29T11:11:47.383 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-29T11:11:47.383 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-29T11:11:47.460 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-29T11:11:47.523 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-29T11:11:47.556 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:11:47.634 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:11:47.967 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:11:47.988 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:11:48.231 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:11:48.231 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:11:48.231 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-29T11:11:48.231 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:11:48.231 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:11:48.231 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-29T11:11:48.241 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:11:48.242 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:11:48.242 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-29T11:11:48.242 INFO:teuthology.orchestra.run.smithi047.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:11:48.242 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:11:48.242 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-29T11:11:48.393 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install librados2 2024-03-29T11:11:48.395 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install librados2 2024-03-29T11:11:48.957 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:51 ago on Fri 29 Mar 2024 11:09:57 AM UTC. 2024-03-29T11:11:48.967 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:59 ago on Fri 29 Mar 2024 11:09:49 AM UTC. 2024-03-29T11:11:49.054 INFO:teuthology.orchestra.run.smithi121.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:11:49.067 INFO:teuthology.orchestra.run.smithi047.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:11:49.105 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-29T11:11:49.106 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-03-29T11:11:49.106 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-29T11:11:49.122 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-29T11:11:49.123 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2024-03-29T11:11:49.123 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-29T11:11:49.155 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install librbd1 2024-03-29T11:11:49.171 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install librbd1 2024-03-29T11:11:49.717 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:52 ago on Fri 29 Mar 2024 11:09:57 AM UTC. 2024-03-29T11:11:49.744 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:02:00 ago on Fri 29 Mar 2024 11:09:49 AM UTC. 2024-03-29T11:11:49.813 INFO:teuthology.orchestra.run.smithi121.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:11:49.845 INFO:teuthology.orchestra.run.smithi047.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:11:49.865 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-29T11:11:49.866 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-03-29T11:11:49.866 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-29T11:11:49.900 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-29T11:11:49.901 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2024-03-29T11:11:49.901 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-29T11:11:49.908 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python3-rados 2024-03-29T11:11:49.954 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install python3-rados 2024-03-29T11:11:50.470 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:53 ago on Fri 29 Mar 2024 11:09:57 AM UTC. 2024-03-29T11:11:50.533 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:02:01 ago on Fri 29 Mar 2024 11:09:49 AM UTC. 2024-03-29T11:11:50.568 INFO:teuthology.orchestra.run.smithi121.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:11:50.620 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-29T11:11:50.621 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-03-29T11:11:50.621 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-29T11:11:50.635 INFO:teuthology.orchestra.run.smithi047.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:11:50.671 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python3-rgw 2024-03-29T11:11:50.692 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-29T11:11:50.693 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2024-03-29T11:11:50.693 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-29T11:11:50.749 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install python3-rgw 2024-03-29T11:11:51.240 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:54 ago on Fri 29 Mar 2024 11:09:57 AM UTC. 2024-03-29T11:11:51.328 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:02:02 ago on Fri 29 Mar 2024 11:09:49 AM UTC. 2024-03-29T11:11:51.339 INFO:teuthology.orchestra.run.smithi121.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:11:51.392 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-29T11:11:51.393 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-03-29T11:11:51.393 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-29T11:11:51.430 INFO:teuthology.orchestra.run.smithi047.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:11:51.442 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python3-cephfs 2024-03-29T11:11:51.485 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-29T11:11:51.486 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2024-03-29T11:11:51.486 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-29T11:11:51.551 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install python3-cephfs 2024-03-29T11:11:52.007 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:55 ago on Fri 29 Mar 2024 11:09:57 AM UTC. 2024-03-29T11:11:52.106 INFO:teuthology.orchestra.run.smithi121.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:11:52.132 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:02:03 ago on Fri 29 Mar 2024 11:09:49 AM UTC. 2024-03-29T11:11:52.158 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-29T11:11:52.159 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-03-29T11:11:52.159 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-29T11:11:52.208 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python3-rbd 2024-03-29T11:11:52.234 INFO:teuthology.orchestra.run.smithi047.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:11:52.288 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-29T11:11:52.289 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2024-03-29T11:11:52.289 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-29T11:11:52.349 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install python3-rbd 2024-03-29T11:11:52.769 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:55 ago on Fri 29 Mar 2024 11:09:57 AM UTC. 2024-03-29T11:11:52.867 INFO:teuthology.orchestra.run.smithi121.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:11:52.919 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-29T11:11:52.920 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-03-29T11:11:52.920 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-29T11:11:52.928 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:02:03 ago on Fri 29 Mar 2024 11:09:49 AM UTC. 2024-03-29T11:11:52.962 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install rbd-fuse 2024-03-29T11:11:53.031 INFO:teuthology.orchestra.run.smithi047.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:11:53.088 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-29T11:11:53.089 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2024-03-29T11:11:53.089 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-29T11:11:53.138 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install rbd-fuse 2024-03-29T11:11:53.522 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:56 ago on Fri 29 Mar 2024 11:09:57 AM UTC. 2024-03-29T11:11:53.671 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-29T11:11:53.671 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:11:53.671 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-03-29T11:11:53.672 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:11:53.672 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-29T11:11:53.672 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:11:53.672 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:11:53.672 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-29T11:11:53.672 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:11:53.672 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-03-29T11:11:53.672 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:11:53.672 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 85 k 2024-03-29T11:11:53.672 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 231 k 2024-03-29T11:11:53.672 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-29T11:11:53.716 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:02:04 ago on Fri 29 Mar 2024 11:09:49 AM UTC. 2024-03-29T11:11:53.871 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-29T11:11:53.871 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-29T11:11:53.871 INFO:teuthology.orchestra.run.smithi047.stdout: Package Architecture Version Repository Size 2024-03-29T11:11:53.872 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-29T11:11:53.872 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-29T11:11:53.872 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:11:53.872 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:11:53.872 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-29T11:11:53.872 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-29T11:11:53.872 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-29T11:11:53.872 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:11:53.872 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 85 k 2024-03-29T11:11:53.872 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 231 k 2024-03-29T11:11:53.872 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-29T11:11:54.002 INFO:teuthology.orchestra.run.smithi121.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 259 kB/s | 85 kB 00:00 2024-03-29T11:11:54.002 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:11:54.003 INFO:teuthology.orchestra.run.smithi121.stdout:Total 258 kB/s | 85 kB 00:00 2024-03-29T11:11:54.003 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-29T11:11:54.010 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-29T11:11:54.010 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-29T11:11:54.066 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-29T11:11:54.066 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-29T11:11:54.086 INFO:teuthology.orchestra.run.smithi047.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 399 kB/s | 85 kB 00:00 2024-03-29T11:11:54.087 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:11:54.087 INFO:teuthology.orchestra.run.smithi047.stdout:Total 397 kB/s | 85 kB 00:00 2024-03-29T11:11:54.087 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-29T11:11:54.094 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-29T11:11:54.095 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-29T11:11:54.153 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-29T11:11:54.154 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-29T11:11:54.209 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-29T11:11:54.295 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-29T11:11:54.309 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:11:54.387 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:11:54.712 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:11:54.771 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:11:55.021 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:11:55.021 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:11:55.021 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-29T11:11:55.021 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:11:55.021 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:11:55.021 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-29T11:11:55.041 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:11:55.041 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:11:55.042 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-29T11:11:55.042 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:11:55.042 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:11:55.042 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-29T11:11:55.245 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install rbd-mirror 2024-03-29T11:11:55.271 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install rbd-mirror 2024-03-29T11:11:55.825 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:58 ago on Fri 29 Mar 2024 11:09:57 AM UTC. 2024-03-29T11:11:55.846 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:02:06 ago on Fri 29 Mar 2024 11:09:49 AM UTC. 2024-03-29T11:11:55.980 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-29T11:11:55.981 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:11:55.981 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-03-29T11:11:55.981 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:11:55.981 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-29T11:11:55.981 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:11:55.981 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:11:55.982 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-29T11:11:55.982 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:11:55.982 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-03-29T11:11:55.982 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:11:55.982 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 3.1 M 2024-03-29T11:11:55.982 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 13 M 2024-03-29T11:11:55.982 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-29T11:11:56.004 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-29T11:11:56.005 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-29T11:11:56.005 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-03-29T11:11:56.005 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-29T11:11:56.005 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-29T11:11:56.006 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:11:56.006 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:11:56.006 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-29T11:11:56.006 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-29T11:11:56.006 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-29T11:11:56.006 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:11:56.006 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 3.1 M 2024-03-29T11:11:56.006 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 13 M 2024-03-29T11:11:56.006 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-29T11:11:57.054 INFO:teuthology.orchestra.run.smithi121.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 2.9 MB/s | 3.1 MB 00:01 2024-03-29T11:11:57.055 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:11:57.055 INFO:teuthology.orchestra.run.smithi121.stdout:Total 2.9 MB/s | 3.1 MB 00:01 2024-03-29T11:11:57.055 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-29T11:11:57.065 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-29T11:11:57.065 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-29T11:11:57.073 INFO:teuthology.orchestra.run.smithi047.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 2.9 MB/s | 3.1 MB 00:01 2024-03-29T11:11:57.074 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:11:57.074 INFO:teuthology.orchestra.run.smithi047.stdout:Total 2.9 MB/s | 3.1 MB 00:01 2024-03-29T11:11:57.074 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-29T11:11:57.084 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-29T11:11:57.085 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-29T11:11:57.139 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-29T11:11:57.139 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-29T11:11:57.163 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-29T11:11:57.164 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-29T11:11:57.574 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-29T11:11:57.610 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-29T11:11:57.647 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:11:57.672 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:11:57.672 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:11:57.672 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:11:57.672 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:11:57.672 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:11:57.672 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:11:57.682 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:11:57.707 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:11:57.707 INFO:teuthology.orchestra.run.smithi047.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:11:57.707 INFO:teuthology.orchestra.run.smithi047.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:11:57.707 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:11:57.708 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:11:57.708 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:11:58.822 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:11:58.822 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:11:58.822 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-29T11:11:58.822 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:11:58.823 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:11:58.823 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-29T11:11:58.823 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:11:58.823 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:11:58.824 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-29T11:11:58.824 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:11:58.824 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:11:58.824 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-29T11:11:59.058 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install rbd-nbd 2024-03-29T11:11:59.072 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install rbd-nbd 2024-03-29T11:11:59.636 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:02 ago on Fri 29 Mar 2024 11:09:57 AM UTC. 2024-03-29T11:11:59.649 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:02:10 ago on Fri 29 Mar 2024 11:09:49 AM UTC. 2024-03-29T11:11:59.786 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-29T11:11:59.786 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:11:59.786 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-03-29T11:11:59.787 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:11:59.787 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-29T11:11:59.787 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:11:59.787 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:11:59.789 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-29T11:11:59.789 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-03-29T11:11:59.789 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-03-29T11:11:59.789 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:11:59.789 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 171 k 2024-03-29T11:11:59.789 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 494 k 2024-03-29T11:11:59.790 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-29T11:11:59.811 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-29T11:11:59.813 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-29T11:11:59.813 INFO:teuthology.orchestra.run.smithi047.stdout: Package Architecture Version Repository Size 2024-03-29T11:11:59.813 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-29T11:11:59.813 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-29T11:11:59.813 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:11:59.813 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:11:59.813 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-29T11:11:59.813 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-29T11:11:59.814 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-29T11:11:59.814 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:11:59.814 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 171 k 2024-03-29T11:11:59.814 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 494 k 2024-03-29T11:11:59.814 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-29T11:12:00.124 INFO:teuthology.orchestra.run.smithi121.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 509 kB/s | 171 kB 00:00 2024-03-29T11:12:00.125 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:12:00.125 INFO:teuthology.orchestra.run.smithi121.stdout:Total 507 kB/s | 171 kB 00:00 2024-03-29T11:12:00.125 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-29T11:12:00.134 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-29T11:12:00.135 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-29T11:12:00.159 INFO:teuthology.orchestra.run.smithi047.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 495 kB/s | 171 kB 00:00 2024-03-29T11:12:00.160 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:12:00.160 INFO:teuthology.orchestra.run.smithi047.stdout:Total 493 kB/s | 171 kB 00:00 2024-03-29T11:12:00.160 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-29T11:12:00.169 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-29T11:12:00.169 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-29T11:12:00.189 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-29T11:12:00.189 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-29T11:12:00.228 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-29T11:12:00.229 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-29T11:12:00.336 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-29T11:12:00.380 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-29T11:12:00.450 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:12:00.479 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:12:00.870 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:12:00.892 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:12:01.164 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:12:01.164 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:12:01.164 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-29T11:12:01.165 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:12:01.165 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:12:01.165 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-29T11:12:01.187 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:12:01.187 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:12:01.187 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-29T11:12:01.187 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:12:01.188 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:12:01.188 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-29T11:12:01.386 DEBUG:teuthology.parallel:result is None 2024-03-29T11:12:01.406 DEBUG:teuthology.parallel:result is None 2024-03-29T11:12:01.406 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:12:01.627 DEBUG:teuthology.orchestra.run.smithi047:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:12:01.648 INFO:teuthology.orchestra.run.smithi047.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:12:01.649 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:12:01.649 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:12:01.651 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:12:01.871 DEBUG:teuthology.orchestra.run.smithi121:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:12:01.892 INFO:teuthology.orchestra.run.smithi121.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:12:01.892 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:12:01.893 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:12:01.895 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-29T11:12:01.896 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-29T11:12:01.896 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:12:01.927 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-29T11:12:01.927 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:12:01.960 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-29T11:12:01.960 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-29T11:12:01.961 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:12:01.995 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:12:02.074 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-29T11:12:02.075 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:12:02.102 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:12:02.169 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-29T11:12:02.169 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-29T11:12:02.169 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:12:02.197 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:12:02.264 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-29T11:12:02.264 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:12:02.291 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:12:02.357 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-29T11:12:02.358 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-29T11:12:02.358 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:12:02.385 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:12:02.451 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-29T11:12:02.451 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:12:02.479 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:12:02.546 INFO:teuthology.run_tasks:Running task ceph... 2024-03-29T11:12:02.640 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-29T11:12:02.640 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 777 /var/log/ceph 2024-03-29T11:12:02.643 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 777 /var/log/ceph 2024-03-29T11:12:02.669 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-29T11:12:02.669 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:12:02.710 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:12:02.736 INFO:tasks.ceph:Creating extra log directories... 2024-03-29T11:12:02.737 DEBUG:teuthology.orchestra.run.smithi047:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:12:02.777 DEBUG:teuthology.orchestra.run.smithi121:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:12:02.814 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-29T11:12:02.814 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-29T11:12:02.814 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622679', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep}', 'email': None, 'first_in_suite': False, 'job_id': '7622679', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 8152, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi047.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPNhVwhJk+rZpOqy4vRT+1shz5VX6cJK93bgh6UFH/YgmVvQNabahQprKdUnKhgDVpyMN5spp1rihixflu+VieQ=', 'smithi121.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMS5EAtFB8SYN2sAGTqa/rrYY+xvMfYBmHUjp0wioVu/18xT7z+V0ACprXjanFGxqrTGH9apissqBCVGhTCTp84='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-26_05:45:44', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-29T11:12:02.814 DEBUG:teuthology.orchestra.run.smithi047:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:12:02.844 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:12:02.859 DEBUG:teuthology.orchestra.run.smithi047:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:12:02.899 DEBUG:teuthology.orchestra.run.smithi121:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:12:02.926 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-29T11:12:02.926 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:12:02.979 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:12:02.979 DEBUG:teuthology.orchestra.run.smithi047:> stat /dev/vg_nvme/lv_1 2024-03-29T11:12:03.033 INFO:teuthology.orchestra.run.smithi047.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:12:03.033 INFO:teuthology.orchestra.run.smithi047.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:12:03.033 INFO:teuthology.orchestra.run.smithi047.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-29T11:12:03.033 INFO:teuthology.orchestra.run.smithi047.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:12:03.033 INFO:teuthology.orchestra.run.smithi047.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:12:03.033 INFO:teuthology.orchestra.run.smithi047.stdout:Access: 2024-03-29 11:11:57.973027234 +0000 2024-03-29T11:12:03.033 INFO:teuthology.orchestra.run.smithi047.stdout:Modify: 2024-03-29 11:09:07.433572147 +0000 2024-03-29T11:12:03.034 INFO:teuthology.orchestra.run.smithi047.stdout:Change: 2024-03-29 11:09:07.433572147 +0000 2024-03-29T11:12:03.034 INFO:teuthology.orchestra.run.smithi047.stdout: Birth: 2024-03-29 11:09:07.433572147 +0000 2024-03-29T11:12:03.034 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:12:03.098 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records in 2024-03-29T11:12:03.098 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records out 2024-03-29T11:12:03.098 INFO:teuthology.orchestra.run.smithi047.stderr:512 bytes copied, 8.6442e-05 s, 5.9 MB/s 2024-03-29T11:12:03.099 DEBUG:teuthology.orchestra.run.smithi047:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:12:03.154 DEBUG:teuthology.orchestra.run.smithi047:> stat /dev/vg_nvme/lv_2 2024-03-29T11:12:03.209 INFO:teuthology.orchestra.run.smithi047.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:12:03.209 INFO:teuthology.orchestra.run.smithi047.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:12:03.209 INFO:teuthology.orchestra.run.smithi047.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-29T11:12:03.209 INFO:teuthology.orchestra.run.smithi047.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:12:03.209 INFO:teuthology.orchestra.run.smithi047.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:12:03.209 INFO:teuthology.orchestra.run.smithi047.stdout:Access: 2024-03-29 11:11:57.973027234 +0000 2024-03-29T11:12:03.209 INFO:teuthology.orchestra.run.smithi047.stdout:Modify: 2024-03-29 11:09:07.764564961 +0000 2024-03-29T11:12:03.209 INFO:teuthology.orchestra.run.smithi047.stdout:Change: 2024-03-29 11:09:07.764564961 +0000 2024-03-29T11:12:03.209 INFO:teuthology.orchestra.run.smithi047.stdout: Birth: 2024-03-29 11:09:07.764564961 +0000 2024-03-29T11:12:03.210 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:12:03.274 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records in 2024-03-29T11:12:03.274 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records out 2024-03-29T11:12:03.274 INFO:teuthology.orchestra.run.smithi047.stderr:512 bytes copied, 8.0434e-05 s, 6.4 MB/s 2024-03-29T11:12:03.278 DEBUG:teuthology.orchestra.run.smithi047:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:12:03.334 DEBUG:teuthology.orchestra.run.smithi047:> stat /dev/vg_nvme/lv_3 2024-03-29T11:12:03.388 INFO:teuthology.orchestra.run.smithi047.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:12:03.388 INFO:teuthology.orchestra.run.smithi047.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:12:03.388 INFO:teuthology.orchestra.run.smithi047.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-29T11:12:03.388 INFO:teuthology.orchestra.run.smithi047.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:12:03.388 INFO:teuthology.orchestra.run.smithi047.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:12:03.388 INFO:teuthology.orchestra.run.smithi047.stdout:Access: 2024-03-29 11:11:57.973027234 +0000 2024-03-29T11:12:03.388 INFO:teuthology.orchestra.run.smithi047.stdout:Modify: 2024-03-29 11:09:08.101557644 +0000 2024-03-29T11:12:03.388 INFO:teuthology.orchestra.run.smithi047.stdout:Change: 2024-03-29 11:09:08.101557644 +0000 2024-03-29T11:12:03.389 INFO:teuthology.orchestra.run.smithi047.stdout: Birth: 2024-03-29 11:09:08.101557644 +0000 2024-03-29T11:12:03.389 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:12:03.453 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records in 2024-03-29T11:12:03.454 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records out 2024-03-29T11:12:03.454 INFO:teuthology.orchestra.run.smithi047.stderr:512 bytes copied, 8.8154e-05 s, 5.8 MB/s 2024-03-29T11:12:03.455 DEBUG:teuthology.orchestra.run.smithi047:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:12:03.510 DEBUG:teuthology.orchestra.run.smithi047:> stat /dev/vg_nvme/lv_4 2024-03-29T11:12:03.564 INFO:teuthology.orchestra.run.smithi047.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:12:03.564 INFO:teuthology.orchestra.run.smithi047.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:12:03.564 INFO:teuthology.orchestra.run.smithi047.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-29T11:12:03.564 INFO:teuthology.orchestra.run.smithi047.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:12:03.564 INFO:teuthology.orchestra.run.smithi047.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:12:03.565 INFO:teuthology.orchestra.run.smithi047.stdout:Access: 2024-03-29 11:11:57.974027212 +0000 2024-03-29T11:12:03.565 INFO:teuthology.orchestra.run.smithi047.stdout:Modify: 2024-03-29 11:09:08.423550653 +0000 2024-03-29T11:12:03.565 INFO:teuthology.orchestra.run.smithi047.stdout:Change: 2024-03-29 11:09:08.423550653 +0000 2024-03-29T11:12:03.565 INFO:teuthology.orchestra.run.smithi047.stdout: Birth: 2024-03-29 11:09:08.423550653 +0000 2024-03-29T11:12:03.565 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:12:03.629 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records in 2024-03-29T11:12:03.630 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records out 2024-03-29T11:12:03.630 INFO:teuthology.orchestra.run.smithi047.stderr:512 bytes copied, 9.4341e-05 s, 5.4 MB/s 2024-03-29T11:12:03.631 DEBUG:teuthology.orchestra.run.smithi047:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:12:03.686 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:12:03.686 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-29T11:12:03.686 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:12:03.703 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:12:03.704 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_1 2024-03-29T11:12:03.761 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:12:03.761 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:12:03.761 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-29T11:12:03.761 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:12:03.761 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:12:03.761 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-03-29 11:11:57.942434106 +0000 2024-03-29T11:12:03.761 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-03-29 11:09:05.413172851 +0000 2024-03-29T11:12:03.761 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-03-29 11:09:05.413172851 +0000 2024-03-29T11:12:03.761 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: 2024-03-29 11:09:05.413172851 +0000 2024-03-29T11:12:03.762 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:12:03.829 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-03-29T11:12:03.829 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-03-29T11:12:03.829 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000268806 s, 1.9 MB/s 2024-03-29T11:12:03.830 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:12:03.890 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_2 2024-03-29T11:12:03.945 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:12:03.946 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:12:03.946 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-29T11:12:03.946 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:12:03.946 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:12:03.946 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-03-29 11:11:57.943434093 +0000 2024-03-29T11:12:03.946 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-03-29 11:09:05.763168233 +0000 2024-03-29T11:12:03.946 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-03-29 11:09:05.763168233 +0000 2024-03-29T11:12:03.946 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: 2024-03-29 11:09:05.763168233 +0000 2024-03-29T11:12:03.946 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:12:04.013 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-03-29T11:12:04.013 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-03-29T11:12:04.013 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000212228 s, 2.4 MB/s 2024-03-29T11:12:04.014 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:12:04.071 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_3 2024-03-29T11:12:04.126 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:12:04.126 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:12:04.126 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-29T11:12:04.126 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:12:04.126 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:12:04.126 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-03-29 11:11:57.943434093 +0000 2024-03-29T11:12:04.126 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-03-29 11:09:06.091163905 +0000 2024-03-29T11:12:04.126 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-03-29 11:09:06.091163905 +0000 2024-03-29T11:12:04.127 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: 2024-03-29 11:09:06.091163905 +0000 2024-03-29T11:12:04.127 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:12:04.191 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-03-29T11:12:04.191 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-03-29T11:12:04.192 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000197189 s, 2.6 MB/s 2024-03-29T11:12:04.192 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:12:04.248 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_4 2024-03-29T11:12:04.303 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:12:04.304 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:12:04.304 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-29T11:12:04.304 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:12:04.304 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:12:04.304 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-03-29 11:11:57.943434093 +0000 2024-03-29T11:12:04.304 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-03-29 11:09:06.423159524 +0000 2024-03-29T11:12:04.304 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-03-29 11:09:06.423159524 +0000 2024-03-29T11:12:04.304 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: 2024-03-29 11:09:06.423159524 +0000 2024-03-29T11:12:04.304 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:12:04.369 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-03-29T11:12:04.369 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-03-29T11:12:04.369 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000201404 s, 2.5 MB/s 2024-03-29T11:12:04.371 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:12:04.426 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:12:04.426 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi047.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi121.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:12:04.426 INFO:tasks.ceph:Generating config... 2024-03-29T11:12:04.427 INFO:tasks.ceph:[client] debug monc = 20 2024-03-29T11:12:04.427 INFO:tasks.ceph:[client] debug ms = 1 2024-03-29T11:12:04.427 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-29T11:12:04.428 INFO:tasks.ceph:[client] debug rados = 20 2024-03-29T11:12:04.428 INFO:tasks.ceph:[global] lockdep = True 2024-03-29T11:12:04.428 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-29T11:12:04.428 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-03-29T11:12:04.428 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-29T11:12:04.428 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-29T11:12:04.428 INFO:tasks.ceph:[global] ms type = async 2024-03-29T11:12:04.428 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-29T11:12:04.428 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-29T11:12:04.428 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-29T11:12:04.428 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-29T11:12:04.428 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-29T11:12:04.428 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-29T11:12:04.428 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-29T11:12:04.428 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-29T11:12:04.429 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-29T11:12:04.429 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-29T11:12:04.429 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-29T11:12:04.429 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-29T11:12:04.429 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-29T11:12:04.429 INFO:tasks.ceph:[osd] bluestore compression algorithm = zstd 2024-03-29T11:12:04.429 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-29T11:12:04.429 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-29T11:12:04.429 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-29T11:12:04.429 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-29T11:12:04.429 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-29T11:12:04.429 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-29T11:12:04.429 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-29T11:12:04.429 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-29T11:12:04.429 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-29T11:12:04.429 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-29T11:12:04.430 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-29T11:12:04.430 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-29T11:12:04.430 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-29T11:12:04.430 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-29T11:12:04.430 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-29T11:12:04.430 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-29T11:12:04.430 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-29T11:12:04.430 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-29T11:12:04.430 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-29T11:12:04.430 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-29T11:12:04.430 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-29T11:12:04.430 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-29T11:12:04.431 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-29T11:12:04.431 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-29T11:12:04.431 INFO:tasks.ceph:Setting up mon.a... 2024-03-29T11:12:04.431 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-29T11:12:04.492 INFO:teuthology.orchestra.run.smithi047.stdout:creating /etc/ceph/ceph.keyring 2024-03-29T11:12:04.494 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-29T11:12:04.542 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:12:04.569 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.47:6789]'), ('mon.c', '[v1:172.21.15.47:6790]'), ('mon.b', '[v1:172.21.15.121:6789]')] 2024-03-29T11:12:04.570 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.47:6789],[v1:172.21.15.47:6790],[v1:172.21.15.121:6789]', 'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-29T11:12:04.571 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-29T11:12:04.571 DEBUG:teuthology.orchestra.run.smithi047:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:12:04.625 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.47:6789]' --addv c '[v1:172.21.15.47:6790]' --addv b '[v1:172.21.15.121:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:12:04.711 INFO:teuthology.orchestra.run.smithi047.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:12:04.712 INFO:teuthology.orchestra.run.smithi047.stdout:monmaptool: generated fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:04.712 INFO:teuthology.orchestra.run.smithi047.stdout:setting min_mon_release = quincy 2024-03-29T11:12:04.712 INFO:teuthology.orchestra.run.smithi047.stdout:epoch 0 2024-03-29T11:12:04.712 INFO:teuthology.orchestra.run.smithi047.stdout:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:04.712 INFO:teuthology.orchestra.run.smithi047.stdout:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:04.712 INFO:teuthology.orchestra.run.smithi047.stdout:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:04.712 INFO:teuthology.orchestra.run.smithi047.stdout:min_mon_release 17 (quincy) 2024-03-29T11:12:04.712 INFO:teuthology.orchestra.run.smithi047.stdout:election_strategy: 1 2024-03-29T11:12:04.712 INFO:teuthology.orchestra.run.smithi047.stdout:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:04.712 INFO:teuthology.orchestra.run.smithi047.stdout:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:04.712 INFO:teuthology.orchestra.run.smithi047.stdout:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:04.713 INFO:teuthology.orchestra.run.smithi047.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-29T11:12:04.714 DEBUG:teuthology.orchestra.run.smithi047:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:12:04.730 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID bb938893-5bcf-496d-8a49-24fbf9065510... 2024-03-29T11:12:04.731 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:12:04.772 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:12:04.820 INFO:teuthology.orchestra.run.smithi047.stdout:[global] 2024-03-29T11:12:04.821 INFO:teuthology.orchestra.run.smithi047.stdout: chdir = "" 2024-03-29T11:12:04.821 INFO:teuthology.orchestra.run.smithi047.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:12:04.821 INFO:teuthology.orchestra.run.smithi047.stdout: auth supported = cephx 2024-03-29T11:12:04.821 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:12:04.821 INFO:teuthology.orchestra.run.smithi047.stdout: filestore xattr use omap = true 2024-03-29T11:12:04.821 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:12:04.821 INFO:teuthology.orchestra.run.smithi047.stdout: mon clock drift allowed = 1.000 2024-03-29T11:12:04.821 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:12:04.821 INFO:teuthology.orchestra.run.smithi047.stdout: osd crush chooseleaf type = 0 2024-03-29T11:12:04.821 INFO:teuthology.orchestra.run.smithi047.stdout: auth debug = true 2024-03-29T11:12:04.821 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:12:04.822 INFO:teuthology.orchestra.run.smithi047.stdout: ms die on old message = true 2024-03-29T11:12:04.822 INFO:teuthology.orchestra.run.smithi047.stdout: ms die on bug = true 2024-03-29T11:12:04.822 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:12:04.822 INFO:teuthology.orchestra.run.smithi047.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:12:04.822 INFO:teuthology.orchestra.run.smithi047.stdout: mon pg warn max object skew = 0 2024-03-29T11:12:04.822 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:12:04.822 INFO:teuthology.orchestra.run.smithi047.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:12:04.822 INFO:teuthology.orchestra.run.smithi047.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:12:04.822 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:12:04.822 INFO:teuthology.orchestra.run.smithi047.stdout: osd pool default size = 2 2024-03-29T11:12:04.822 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:12:04.822 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd allow primary affinity = true 2024-03-29T11:12:04.822 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd allow pg remap = true 2024-03-29T11:12:04.823 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:12:04.823 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:12:04.823 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on no sortbitwise = false 2024-03-29T11:12:04.823 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:12:04.823 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on too few osds = false 2024-03-29T11:12:04.823 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:12:04.823 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:12:04.823 INFO:teuthology.orchestra.run.smithi047.stdout: mon_allow_pool_size_one = true 2024-03-29T11:12:04.823 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:12:04.823 INFO:teuthology.orchestra.run.smithi047.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:12:04.823 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:12:04.823 INFO:teuthology.orchestra.run.smithi047.stdout: osd default data pool replay window = 5 2024-03-29T11:12:04.823 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:12:04.823 INFO:teuthology.orchestra.run.smithi047.stdout: mon allow pool delete = true 2024-03-29T11:12:04.824 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:12:04.824 INFO:teuthology.orchestra.run.smithi047.stdout: mon cluster log file level = debug 2024-03-29T11:12:04.824 INFO:teuthology.orchestra.run.smithi047.stdout: debug asserts on shutdown = true 2024-03-29T11:12:04.824 INFO:teuthology.orchestra.run.smithi047.stdout: mon health detail to clog = false 2024-03-29T11:12:04.824 INFO:teuthology.orchestra.run.smithi047.stdout: mon host = "[v1:172.21.15.47:6789],[v1:172.21.15.47:6790],[v1:172.21.15.121:6789]" 2024-03-29T11:12:04.824 INFO:teuthology.orchestra.run.smithi047.stdout: lockdep = True 2024-03-29T11:12:04.824 INFO:teuthology.orchestra.run.smithi047.stdout: mon client directed command retry = 5 2024-03-29T11:12:04.824 INFO:teuthology.orchestra.run.smithi047.stdout: mon election default strategy = 1 2024-03-29T11:12:04.824 INFO:teuthology.orchestra.run.smithi047.stdout: ms bind msgr2 = False 2024-03-29T11:12:04.824 INFO:teuthology.orchestra.run.smithi047.stdout: ms inject socket failures = 5000 2024-03-29T11:12:04.824 INFO:teuthology.orchestra.run.smithi047.stdout: ms type = async 2024-03-29T11:12:04.824 INFO:teuthology.orchestra.run.smithi047.stdout: fsid = bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:04.824 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:12:04.825 INFO:teuthology.orchestra.run.smithi047.stdout:[osd] 2024-03-29T11:12:04.825 INFO:teuthology.orchestra.run.smithi047.stdout: osd journal size = 100 2024-03-29T11:12:04.825 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:12:04.825 INFO:teuthology.orchestra.run.smithi047.stdout: osd scrub load threshold = 5.0 2024-03-29T11:12:04.825 INFO:teuthology.orchestra.run.smithi047.stdout: osd scrub max interval = 600 2024-03-29T11:12:04.825 INFO:teuthology.orchestra.run.smithi047.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:12:04.825 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:12:04.825 INFO:teuthology.orchestra.run.smithi047.stdout: osd recover clone overlap = true 2024-03-29T11:12:04.825 INFO:teuthology.orchestra.run.smithi047.stdout: osd recovery max chunk = 1048576 2024-03-29T11:12:04.826 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:12:04.826 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug shutdown = true 2024-03-29T11:12:04.826 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug op order = true 2024-03-29T11:12:04.826 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug verify stray on activate = true 2024-03-29T11:12:04.826 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug trim objects = true 2024-03-29T11:12:04.826 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:12:04.826 INFO:teuthology.orchestra.run.smithi047.stdout: osd open classes on start = true 2024-03-29T11:12:04.826 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug pg log writeout = true 2024-03-29T11:12:04.826 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:12:04.826 INFO:teuthology.orchestra.run.smithi047.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:12:04.826 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:12:04.827 INFO:teuthology.orchestra.run.smithi047.stdout: osd map max advance = 10 2024-03-29T11:12:04.827 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:12:04.827 INFO:teuthology.orchestra.run.smithi047.stdout: journal zero on create = true 2024-03-29T11:12:04.827 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:12:04.827 INFO:teuthology.orchestra.run.smithi047.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:12:04.827 INFO:teuthology.orchestra.run.smithi047.stdout: filestore apply finisher threads = 3 2024-03-29T11:12:04.827 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:12:04.827 INFO:teuthology.orchestra.run.smithi047.stdout: bdev debug aio = true 2024-03-29T11:12:04.827 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug misdirected ops = true 2024-03-29T11:12:04.827 INFO:teuthology.orchestra.run.smithi047.stdout: bluestore block size = 96636764160 2024-03-29T11:12:04.827 INFO:teuthology.orchestra.run.smithi047.stdout: bluestore compression algorithm = zstd 2024-03-29T11:12:04.827 INFO:teuthology.orchestra.run.smithi047.stdout: bluestore compression mode = aggressive 2024-03-29T11:12:04.827 INFO:teuthology.orchestra.run.smithi047.stdout: bluestore fsck on mount = True 2024-03-29T11:12:04.827 INFO:teuthology.orchestra.run.smithi047.stdout: bluestore zero block detection = True 2024-03-29T11:12:04.827 INFO:teuthology.orchestra.run.smithi047.stdout: debug bluefs = 20 2024-03-29T11:12:04.828 INFO:teuthology.orchestra.run.smithi047.stdout: debug bluestore = 20 2024-03-29T11:12:04.828 INFO:teuthology.orchestra.run.smithi047.stdout: debug monc = 20 2024-03-29T11:12:04.828 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2024-03-29T11:12:04.828 INFO:teuthology.orchestra.run.smithi047.stdout: debug osd = 20 2024-03-29T11:12:04.828 INFO:teuthology.orchestra.run.smithi047.stdout: debug rocksdb = 10 2024-03-29T11:12:04.828 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:12:04.828 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd full ratio = 0.9 2024-03-29T11:12:04.828 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:12:04.828 INFO:teuthology.orchestra.run.smithi047.stdout: osd class default list = * 2024-03-29T11:12:04.828 INFO:teuthology.orchestra.run.smithi047.stdout: osd class load list = * 2024-03-29T11:12:04.828 INFO:teuthology.orchestra.run.smithi047.stdout: osd client watch timeout = 120 2024-03-29T11:12:04.828 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug verify cached snaps = True 2024-03-29T11:12:04.828 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug verify missing on start = True 2024-03-29T11:12:04.828 INFO:teuthology.orchestra.run.smithi047.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:12:04.828 INFO:teuthology.orchestra.run.smithi047.stdout: osd mclock override recovery settings = True 2024-03-29T11:12:04.828 INFO:teuthology.orchestra.run.smithi047.stdout: osd objectstore = bluestore 2024-03-29T11:12:04.829 INFO:teuthology.orchestra.run.smithi047.stdout: osd op queue = debug_random 2024-03-29T11:12:04.829 INFO:teuthology.orchestra.run.smithi047.stdout: osd op queue cut off = debug_random 2024-03-29T11:12:04.829 INFO:teuthology.orchestra.run.smithi047.stdout: osd shutdown pgref assert = True 2024-03-29T11:12:04.829 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:12:04.829 INFO:teuthology.orchestra.run.smithi047.stdout:[mgr] 2024-03-29T11:12:04.829 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2024-03-29T11:12:04.829 INFO:teuthology.orchestra.run.smithi047.stdout: debug mgr = 20 2024-03-29T11:12:04.829 INFO:teuthology.orchestra.run.smithi047.stdout: debug mon = 20 2024-03-29T11:12:04.829 INFO:teuthology.orchestra.run.smithi047.stdout: debug auth = 20 2024-03-29T11:12:04.829 INFO:teuthology.orchestra.run.smithi047.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:12:04.829 INFO:teuthology.orchestra.run.smithi047.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:12:04.829 INFO:teuthology.orchestra.run.smithi047.stdout: mgr/telemetry/nag = false 2024-03-29T11:12:04.829 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:12:04.829 INFO:teuthology.orchestra.run.smithi047.stdout:[mon] 2024-03-29T11:12:04.829 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2024-03-29T11:12:04.830 INFO:teuthology.orchestra.run.smithi047.stdout: debug mon = 20 2024-03-29T11:12:04.830 INFO:teuthology.orchestra.run.smithi047.stdout: debug paxos = 20 2024-03-29T11:12:04.830 INFO:teuthology.orchestra.run.smithi047.stdout: debug auth = 20 2024-03-29T11:12:04.830 INFO:teuthology.orchestra.run.smithi047.stdout: mon data avail warn = 5 2024-03-29T11:12:04.830 INFO:teuthology.orchestra.run.smithi047.stdout: mon mgr mkfs grace = 240 2024-03-29T11:12:04.830 INFO:teuthology.orchestra.run.smithi047.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:12:04.830 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd reporter subtree level = osd 2024-03-29T11:12:04.830 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd prime pg temp = true 2024-03-29T11:12:04.830 INFO:teuthology.orchestra.run.smithi047.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:12:04.830 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:12:04.830 INFO:teuthology.orchestra.run.smithi047.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:12:04.830 INFO:teuthology.orchestra.run.smithi047.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:12:04.830 INFO:teuthology.orchestra.run.smithi047.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:12:04.830 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:12:04.831 INFO:teuthology.orchestra.run.smithi047.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:12:04.831 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:12:04.831 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:12:04.831 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:12:04.831 INFO:teuthology.orchestra.run.smithi047.stdout: # 1m isn't quite enough 2024-03-29T11:12:04.831 INFO:teuthology.orchestra.run.smithi047.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:12:04.831 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:12:04.831 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:12:04.831 INFO:teuthology.orchestra.run.smithi047.stdout: mon min osdmap epochs = 50 2024-03-29T11:12:04.832 INFO:teuthology.orchestra.run.smithi047.stdout: mon osdmap full prune interval = 2 2024-03-29T11:12:04.832 INFO:teuthology.orchestra.run.smithi047.stdout: mon osdmap full prune min = 15 2024-03-29T11:12:04.832 INFO:teuthology.orchestra.run.smithi047.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:12:04.832 INFO:teuthology.orchestra.run.smithi047.stdout: mon scrub interval = 300 2024-03-29T11:12:04.832 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on pool no app = False 2024-03-29T11:12:04.832 INFO:teuthology.orchestra.run.smithi047.stdout: paxos service trim min = 10 2024-03-29T11:12:04.832 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:12:04.832 INFO:teuthology.orchestra.run.smithi047.stdout:[client] 2024-03-29T11:12:04.832 INFO:teuthology.orchestra.run.smithi047.stdout: rgw cache enabled = true 2024-03-29T11:12:04.832 INFO:teuthology.orchestra.run.smithi047.stdout: rgw enable ops log = true 2024-03-29T11:12:04.832 INFO:teuthology.orchestra.run.smithi047.stdout: rgw enable usage log = true 2024-03-29T11:12:04.833 INFO:teuthology.orchestra.run.smithi047.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:12:04.833 INFO:teuthology.orchestra.run.smithi047.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:12:04.833 INFO:teuthology.orchestra.run.smithi047.stdout: debug monc = 20 2024-03-29T11:12:04.833 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2024-03-29T11:12:04.833 INFO:teuthology.orchestra.run.smithi047.stdout: debug objecter = 20 2024-03-29T11:12:04.833 INFO:teuthology.orchestra.run.smithi047.stdout: debug rados = 20 2024-03-29T11:12:04.833 INFO:teuthology.orchestra.run.smithi047.stdout:[mon.a] 2024-03-29T11:12:04.833 INFO:teuthology.orchestra.run.smithi047.stdout:[mon.c] 2024-03-29T11:12:04.833 INFO:teuthology.orchestra.run.smithi047.stdout:[mon.b] 2024-03-29T11:12:04.834 INFO:teuthology.orchestra.run.smithi121.stdout:[global] 2024-03-29T11:12:04.834 INFO:teuthology.orchestra.run.smithi121.stdout: chdir = "" 2024-03-29T11:12:04.835 INFO:teuthology.orchestra.run.smithi121.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:12:04.835 INFO:teuthology.orchestra.run.smithi121.stdout: auth supported = cephx 2024-03-29T11:12:04.835 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:12:04.835 INFO:teuthology.orchestra.run.smithi121.stdout: filestore xattr use omap = true 2024-03-29T11:12:04.835 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:12:04.835 INFO:teuthology.orchestra.run.smithi121.stdout: mon clock drift allowed = 1.000 2024-03-29T11:12:04.835 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:12:04.835 INFO:teuthology.orchestra.run.smithi121.stdout: osd crush chooseleaf type = 0 2024-03-29T11:12:04.835 INFO:teuthology.orchestra.run.smithi121.stdout: auth debug = true 2024-03-29T11:12:04.836 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:12:04.836 INFO:teuthology.orchestra.run.smithi121.stdout: ms die on old message = true 2024-03-29T11:12:04.836 INFO:teuthology.orchestra.run.smithi121.stdout: ms die on bug = true 2024-03-29T11:12:04.836 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:12:04.836 INFO:teuthology.orchestra.run.smithi121.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:12:04.836 INFO:teuthology.orchestra.run.smithi121.stdout: mon pg warn max object skew = 0 2024-03-29T11:12:04.836 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:12:04.836 INFO:teuthology.orchestra.run.smithi121.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:12:04.836 INFO:teuthology.orchestra.run.smithi121.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:12:04.836 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:12:04.836 INFO:teuthology.orchestra.run.smithi121.stdout: osd pool default size = 2 2024-03-29T11:12:04.836 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:12:04.836 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd allow primary affinity = true 2024-03-29T11:12:04.836 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd allow pg remap = true 2024-03-29T11:12:04.836 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:12:04.837 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:12:04.837 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on no sortbitwise = false 2024-03-29T11:12:04.837 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:12:04.837 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on too few osds = false 2024-03-29T11:12:04.837 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:12:04.837 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:12:04.837 INFO:teuthology.orchestra.run.smithi121.stdout: mon_allow_pool_size_one = true 2024-03-29T11:12:04.837 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:12:04.837 INFO:teuthology.orchestra.run.smithi121.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:12:04.837 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:12:04.837 INFO:teuthology.orchestra.run.smithi121.stdout: osd default data pool replay window = 5 2024-03-29T11:12:04.837 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:12:04.837 INFO:teuthology.orchestra.run.smithi121.stdout: mon allow pool delete = true 2024-03-29T11:12:04.837 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:12:04.837 INFO:teuthology.orchestra.run.smithi121.stdout: mon cluster log file level = debug 2024-03-29T11:12:04.838 INFO:teuthology.orchestra.run.smithi121.stdout: debug asserts on shutdown = true 2024-03-29T11:12:04.838 INFO:teuthology.orchestra.run.smithi121.stdout: mon health detail to clog = false 2024-03-29T11:12:04.838 INFO:teuthology.orchestra.run.smithi121.stdout: mon host = "[v1:172.21.15.47:6789],[v1:172.21.15.47:6790],[v1:172.21.15.121:6789]" 2024-03-29T11:12:04.838 INFO:teuthology.orchestra.run.smithi121.stdout: lockdep = True 2024-03-29T11:12:04.838 INFO:teuthology.orchestra.run.smithi121.stdout: mon client directed command retry = 5 2024-03-29T11:12:04.838 INFO:teuthology.orchestra.run.smithi121.stdout: mon election default strategy = 1 2024-03-29T11:12:04.838 INFO:teuthology.orchestra.run.smithi121.stdout: ms bind msgr2 = False 2024-03-29T11:12:04.838 INFO:teuthology.orchestra.run.smithi121.stdout: ms inject socket failures = 5000 2024-03-29T11:12:04.838 INFO:teuthology.orchestra.run.smithi121.stdout: ms type = async 2024-03-29T11:12:04.838 INFO:teuthology.orchestra.run.smithi121.stdout: fsid = bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:04.838 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:12:04.838 INFO:teuthology.orchestra.run.smithi121.stdout:[osd] 2024-03-29T11:12:04.838 INFO:teuthology.orchestra.run.smithi121.stdout: osd journal size = 100 2024-03-29T11:12:04.838 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:12:04.839 INFO:teuthology.orchestra.run.smithi121.stdout: osd scrub load threshold = 5.0 2024-03-29T11:12:04.839 INFO:teuthology.orchestra.run.smithi121.stdout: osd scrub max interval = 600 2024-03-29T11:12:04.839 INFO:teuthology.orchestra.run.smithi121.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:12:04.839 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:12:04.839 INFO:teuthology.orchestra.run.smithi121.stdout: osd recover clone overlap = true 2024-03-29T11:12:04.839 INFO:teuthology.orchestra.run.smithi121.stdout: osd recovery max chunk = 1048576 2024-03-29T11:12:04.839 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:12:04.839 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug shutdown = true 2024-03-29T11:12:04.839 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug op order = true 2024-03-29T11:12:04.839 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug verify stray on activate = true 2024-03-29T11:12:04.839 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug trim objects = true 2024-03-29T11:12:04.839 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:12:04.839 INFO:teuthology.orchestra.run.smithi121.stdout: osd open classes on start = true 2024-03-29T11:12:04.839 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug pg log writeout = true 2024-03-29T11:12:04.839 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:12:04.839 INFO:teuthology.orchestra.run.smithi121.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:12:04.840 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:12:04.840 INFO:teuthology.orchestra.run.smithi121.stdout: osd map max advance = 10 2024-03-29T11:12:04.840 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:12:04.840 INFO:teuthology.orchestra.run.smithi121.stdout: journal zero on create = true 2024-03-29T11:12:04.840 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:12:04.840 INFO:teuthology.orchestra.run.smithi121.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:12:04.840 INFO:teuthology.orchestra.run.smithi121.stdout: filestore apply finisher threads = 3 2024-03-29T11:12:04.840 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:12:04.840 INFO:teuthology.orchestra.run.smithi121.stdout: bdev debug aio = true 2024-03-29T11:12:04.840 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug misdirected ops = true 2024-03-29T11:12:04.840 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore block size = 96636764160 2024-03-29T11:12:04.840 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore compression algorithm = zstd 2024-03-29T11:12:04.840 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore compression mode = aggressive 2024-03-29T11:12:04.840 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore fsck on mount = True 2024-03-29T11:12:04.840 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore zero block detection = True 2024-03-29T11:12:04.841 INFO:teuthology.orchestra.run.smithi121.stdout: debug bluefs = 20 2024-03-29T11:12:04.841 INFO:teuthology.orchestra.run.smithi121.stdout: debug bluestore = 20 2024-03-29T11:12:04.841 INFO:teuthology.orchestra.run.smithi121.stdout: debug monc = 20 2024-03-29T11:12:04.841 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2024-03-29T11:12:04.841 INFO:teuthology.orchestra.run.smithi121.stdout: debug osd = 20 2024-03-29T11:12:04.841 INFO:teuthology.orchestra.run.smithi121.stdout: debug rocksdb = 10 2024-03-29T11:12:04.841 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:12:04.841 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd full ratio = 0.9 2024-03-29T11:12:04.841 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:12:04.841 INFO:teuthology.orchestra.run.smithi121.stdout: osd class default list = * 2024-03-29T11:12:04.841 INFO:teuthology.orchestra.run.smithi121.stdout: osd class load list = * 2024-03-29T11:12:04.841 INFO:teuthology.orchestra.run.smithi121.stdout: osd client watch timeout = 120 2024-03-29T11:12:04.841 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug verify cached snaps = True 2024-03-29T11:12:04.841 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug verify missing on start = True 2024-03-29T11:12:04.841 INFO:teuthology.orchestra.run.smithi121.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:12:04.842 INFO:teuthology.orchestra.run.smithi121.stdout: osd mclock override recovery settings = True 2024-03-29T11:12:04.842 INFO:teuthology.orchestra.run.smithi121.stdout: osd objectstore = bluestore 2024-03-29T11:12:04.842 INFO:teuthology.orchestra.run.smithi121.stdout: osd op queue = debug_random 2024-03-29T11:12:04.842 INFO:teuthology.orchestra.run.smithi121.stdout: osd op queue cut off = debug_random 2024-03-29T11:12:04.842 INFO:teuthology.orchestra.run.smithi121.stdout: osd shutdown pgref assert = True 2024-03-29T11:12:04.842 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:12:04.842 INFO:teuthology.orchestra.run.smithi121.stdout:[mgr] 2024-03-29T11:12:04.842 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2024-03-29T11:12:04.842 INFO:teuthology.orchestra.run.smithi121.stdout: debug mgr = 20 2024-03-29T11:12:04.842 INFO:teuthology.orchestra.run.smithi121.stdout: debug mon = 20 2024-03-29T11:12:04.842 INFO:teuthology.orchestra.run.smithi121.stdout: debug auth = 20 2024-03-29T11:12:04.842 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:12:04.842 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:12:04.842 INFO:teuthology.orchestra.run.smithi121.stdout: mgr/telemetry/nag = false 2024-03-29T11:12:04.842 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:12:04.843 INFO:teuthology.orchestra.run.smithi121.stdout:[mon] 2024-03-29T11:12:04.843 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2024-03-29T11:12:04.843 INFO:teuthology.orchestra.run.smithi121.stdout: debug mon = 20 2024-03-29T11:12:04.843 INFO:teuthology.orchestra.run.smithi121.stdout: debug paxos = 20 2024-03-29T11:12:04.843 INFO:teuthology.orchestra.run.smithi121.stdout: debug auth = 20 2024-03-29T11:12:04.843 INFO:teuthology.orchestra.run.smithi121.stdout: mon data avail warn = 5 2024-03-29T11:12:04.843 INFO:teuthology.orchestra.run.smithi121.stdout: mon mgr mkfs grace = 240 2024-03-29T11:12:04.843 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:12:04.843 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd reporter subtree level = osd 2024-03-29T11:12:04.843 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd prime pg temp = true 2024-03-29T11:12:04.843 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:12:04.843 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:12:04.843 INFO:teuthology.orchestra.run.smithi121.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:12:04.843 INFO:teuthology.orchestra.run.smithi121.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:12:04.843 INFO:teuthology.orchestra.run.smithi121.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:12:04.843 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:12:04.844 INFO:teuthology.orchestra.run.smithi121.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:12:04.844 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:12:04.844 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:12:04.844 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:12:04.844 INFO:teuthology.orchestra.run.smithi121.stdout: # 1m isn't quite enough 2024-03-29T11:12:04.844 INFO:teuthology.orchestra.run.smithi121.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:12:04.844 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:12:04.844 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:12:04.844 INFO:teuthology.orchestra.run.smithi121.stdout: mon min osdmap epochs = 50 2024-03-29T11:12:04.844 INFO:teuthology.orchestra.run.smithi121.stdout: mon osdmap full prune interval = 2 2024-03-29T11:12:04.844 INFO:teuthology.orchestra.run.smithi121.stdout: mon osdmap full prune min = 15 2024-03-29T11:12:04.844 INFO:teuthology.orchestra.run.smithi121.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:12:04.844 INFO:teuthology.orchestra.run.smithi121.stdout: mon scrub interval = 300 2024-03-29T11:12:04.844 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on pool no app = False 2024-03-29T11:12:04.844 INFO:teuthology.orchestra.run.smithi121.stdout: paxos service trim min = 10 2024-03-29T11:12:04.845 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-29T11:12:04.845 INFO:teuthology.orchestra.run.smithi121.stdout:[client] 2024-03-29T11:12:04.845 INFO:teuthology.orchestra.run.smithi121.stdout: rgw cache enabled = true 2024-03-29T11:12:04.845 INFO:teuthology.orchestra.run.smithi121.stdout: rgw enable ops log = true 2024-03-29T11:12:04.845 INFO:teuthology.orchestra.run.smithi121.stdout: rgw enable usage log = true 2024-03-29T11:12:04.845 INFO:teuthology.orchestra.run.smithi121.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:12:04.845 INFO:teuthology.orchestra.run.smithi121.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:12:04.845 INFO:teuthology.orchestra.run.smithi121.stdout: debug monc = 20 2024-03-29T11:12:04.845 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2024-03-29T11:12:04.845 INFO:teuthology.orchestra.run.smithi121.stdout: debug objecter = 20 2024-03-29T11:12:04.845 INFO:teuthology.orchestra.run.smithi121.stdout: debug rados = 20 2024-03-29T11:12:04.845 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.a] 2024-03-29T11:12:04.845 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.c] 2024-03-29T11:12:04.845 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.b] 2024-03-29T11:12:04.847 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-29T11:12:04.847 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-29T11:12:04.942 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-29T11:12:04.942 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-29T11:12:04.942 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-29T11:12:04.957 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-29T11:12:04.957 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-29T11:12:05.011 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi047.front.sepia.ceph.com 2024-03-29T11:12:05.011 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-29T11:12:05.011 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:12:05.011 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:12:05.089 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-29T11:12:05.090 DEBUG:teuthology.orchestra.run.smithi047:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:12:05.146 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi121.front.sepia.ceph.com 2024-03-29T11:12:05.146 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-29T11:12:05.146 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:12:05.146 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:12:05.186 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-29T11:12:05.186 DEBUG:teuthology.orchestra.run.smithi121:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:12:05.240 INFO:tasks.ceph:Setting up mon nodes... 2024-03-29T11:12:05.240 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-29T11:12:05.241 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:12:05.294 INFO:teuthology.orchestra.run.smithi047.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:12:05.297 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:12:05.359 INFO:teuthology.orchestra.run.smithi121.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:12:05.361 INFO:tasks.ceph:Setting up mds nodes... 2024-03-29T11:12:05.362 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-29T11:12:05.362 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-29T11:12:05.404 INFO:teuthology.orchestra.run.smithi047.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-29T11:12:05.419 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-29T11:12:05.462 INFO:teuthology.orchestra.run.smithi121.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-29T11:12:05.477 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-29T11:12:05.477 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi047.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi121.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:12:05.477 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-29T11:12:05.504 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:12:05.504 INFO:tasks.ceph:role: osd.0 2024-03-29T11:12:05.504 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi047.front.sepia.ceph.com 2024-03-29T11:12:05.504 DEBUG:teuthology.orchestra.run.smithi047:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:12:05.570 INFO:teuthology.orchestra.run.smithi047.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:12:05.570 INFO:teuthology.orchestra.run.smithi047.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:12:05.570 INFO:teuthology.orchestra.run.smithi047.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:12:05.570 INFO:teuthology.orchestra.run.smithi047.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:12:05.570 INFO:teuthology.orchestra.run.smithi047.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:12:05.570 INFO:teuthology.orchestra.run.smithi047.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:12:05.570 INFO:teuthology.orchestra.run.smithi047.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:12:05.570 INFO:teuthology.orchestra.run.smithi047.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:12:05.570 INFO:teuthology.orchestra.run.smithi047.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:12:05.570 INFO:teuthology.orchestra.run.smithi047.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:12:05.602 INFO:teuthology.orchestra.run.smithi047.stdout:Discarding blocks...Done. 2024-03-29T11:12:05.603 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi047.front.sepia.ceph.com -o noatime 2024-03-29T11:12:05.604 DEBUG:teuthology.orchestra.run.smithi047:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-29T11:12:05.675 DEBUG:teuthology.orchestra.run.smithi047:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-29T11:12:05.750 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-29T11:12:05.820 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:12:05.820 INFO:tasks.ceph:role: osd.1 2024-03-29T11:12:05.820 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi047.front.sepia.ceph.com 2024-03-29T11:12:05.820 DEBUG:teuthology.orchestra.run.smithi047:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:12:05.889 INFO:teuthology.orchestra.run.smithi047.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:12:05.889 INFO:teuthology.orchestra.run.smithi047.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:12:05.889 INFO:teuthology.orchestra.run.smithi047.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:12:05.889 INFO:teuthology.orchestra.run.smithi047.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:12:05.890 INFO:teuthology.orchestra.run.smithi047.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:12:05.890 INFO:teuthology.orchestra.run.smithi047.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:12:05.890 INFO:teuthology.orchestra.run.smithi047.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:12:05.890 INFO:teuthology.orchestra.run.smithi047.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:12:05.890 INFO:teuthology.orchestra.run.smithi047.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:12:05.890 INFO:teuthology.orchestra.run.smithi047.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:12:05.924 INFO:teuthology.orchestra.run.smithi047.stdout:Discarding blocks...Done. 2024-03-29T11:12:05.927 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi047.front.sepia.ceph.com -o noatime 2024-03-29T11:12:05.928 DEBUG:teuthology.orchestra.run.smithi047:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-29T11:12:06.018 DEBUG:teuthology.orchestra.run.smithi047:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-29T11:12:06.091 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-29T11:12:06.160 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:12:06.160 INFO:tasks.ceph:role: osd.2 2024-03-29T11:12:06.160 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi047.front.sepia.ceph.com 2024-03-29T11:12:06.160 DEBUG:teuthology.orchestra.run.smithi047:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:12:06.228 INFO:teuthology.orchestra.run.smithi047.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:12:06.228 INFO:teuthology.orchestra.run.smithi047.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:12:06.228 INFO:teuthology.orchestra.run.smithi047.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:12:06.228 INFO:teuthology.orchestra.run.smithi047.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:12:06.228 INFO:teuthology.orchestra.run.smithi047.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:12:06.228 INFO:teuthology.orchestra.run.smithi047.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:12:06.228 INFO:teuthology.orchestra.run.smithi047.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:12:06.229 INFO:teuthology.orchestra.run.smithi047.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:12:06.229 INFO:teuthology.orchestra.run.smithi047.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:12:06.229 INFO:teuthology.orchestra.run.smithi047.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:12:06.260 INFO:teuthology.orchestra.run.smithi047.stdout:Discarding blocks...Done. 2024-03-29T11:12:06.263 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi047.front.sepia.ceph.com -o noatime 2024-03-29T11:12:06.263 DEBUG:teuthology.orchestra.run.smithi047:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-29T11:12:06.342 DEBUG:teuthology.orchestra.run.smithi047:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-29T11:12:06.416 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-29T11:12:06.490 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:12:06.490 INFO:tasks.ceph:role: osd.3 2024-03-29T11:12:06.490 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi047.front.sepia.ceph.com 2024-03-29T11:12:06.490 DEBUG:teuthology.orchestra.run.smithi047:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:12:06.560 INFO:teuthology.orchestra.run.smithi047.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:12:06.560 INFO:teuthology.orchestra.run.smithi047.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:12:06.560 INFO:teuthology.orchestra.run.smithi047.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:12:06.560 INFO:teuthology.orchestra.run.smithi047.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:12:06.560 INFO:teuthology.orchestra.run.smithi047.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:12:06.560 INFO:teuthology.orchestra.run.smithi047.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:12:06.560 INFO:teuthology.orchestra.run.smithi047.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:12:06.560 INFO:teuthology.orchestra.run.smithi047.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:12:06.560 INFO:teuthology.orchestra.run.smithi047.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:12:06.560 INFO:teuthology.orchestra.run.smithi047.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:12:06.593 INFO:teuthology.orchestra.run.smithi047.stdout:Discarding blocks...Done. 2024-03-29T11:12:06.595 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi047.front.sepia.ceph.com -o noatime 2024-03-29T11:12:06.595 DEBUG:teuthology.orchestra.run.smithi047:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-29T11:12:06.667 DEBUG:teuthology.orchestra.run.smithi047:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-29T11:12:06.739 DEBUG:teuthology.orchestra.run.smithi047:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:12:06.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:06.857+0000 7f4f4040d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-29T11:12:06.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:06.858+0000 7f4f4040d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-29T11:12:06.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:06.858+0000 7f4f4040d640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:12:06.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:06.858+0000 7f4f4040d640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:12:06.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:06.858+0000 7f4f4040d640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:12:06.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:06.858+0000 7f4f4040d640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-29T11:12:09.683 DEBUG:teuthology.orchestra.run.smithi047:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-29T11:12:09.714 DEBUG:teuthology.orchestra.run.smithi047:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:12:09.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:09.828+0000 7f3c95cce640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-29T11:12:09.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:09.828+0000 7f3c95cce640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-29T11:12:09.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:09.828+0000 7f3c95cce640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:12:09.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:09.828+0000 7f3c95cce640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:12:09.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:09.828+0000 7f3c95cce640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:12:09.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:09.829+0000 7f3c95cce640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-29T11:12:12.655 DEBUG:teuthology.orchestra.run.smithi047:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-29T11:12:12.687 DEBUG:teuthology.orchestra.run.smithi047:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:12:12.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:12.801+0000 7fc44ce13640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-29T11:12:12.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:12.801+0000 7fc44ce13640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-29T11:12:12.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:12.801+0000 7fc44ce13640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:12:12.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:12.801+0000 7fc44ce13640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:12:12.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:12.801+0000 7fc44ce13640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:12:12.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:12.802+0000 7fc44ce13640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-29T11:12:15.637 DEBUG:teuthology.orchestra.run.smithi047:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-29T11:12:15.668 DEBUG:teuthology.orchestra.run.smithi047:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:12:15.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:15.782+0000 7ff836ebd640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-29T11:12:15.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:15.782+0000 7ff836ebd640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-29T11:12:15.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:15.782+0000 7ff836ebd640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:12:15.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:15.782+0000 7ff836ebd640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:12:15.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:15.782+0000 7ff836ebd640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:12:15.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:15.783+0000 7ff836ebd640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-29T11:12:18.601 DEBUG:teuthology.orchestra.run.smithi047:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-29T11:12:18.632 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-29T11:12:18.664 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:12:18.664 INFO:tasks.ceph:role: osd.4 2024-03-29T11:12:18.664 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi121.front.sepia.ceph.com 2024-03-29T11:12:18.664 DEBUG:teuthology.orchestra.run.smithi121:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:12:18.742 INFO:teuthology.orchestra.run.smithi121.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:12:18.742 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:12:18.742 INFO:teuthology.orchestra.run.smithi121.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:12:18.742 INFO:teuthology.orchestra.run.smithi121.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:12:18.742 INFO:teuthology.orchestra.run.smithi121.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:12:18.742 INFO:teuthology.orchestra.run.smithi121.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:12:18.743 INFO:teuthology.orchestra.run.smithi121.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:12:18.743 INFO:teuthology.orchestra.run.smithi121.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:12:18.743 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:12:18.743 INFO:teuthology.orchestra.run.smithi121.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:12:19.248 INFO:teuthology.orchestra.run.smithi121.stdout:Discarding blocks...Done. 2024-03-29T11:12:19.250 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi121.front.sepia.ceph.com -o noatime 2024-03-29T11:12:19.251 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-29T11:12:19.364 DEBUG:teuthology.orchestra.run.smithi121:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-29T11:12:19.396 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-29T11:12:19.463 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:12:19.463 INFO:tasks.ceph:role: osd.5 2024-03-29T11:12:19.464 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi121.front.sepia.ceph.com 2024-03-29T11:12:19.464 DEBUG:teuthology.orchestra.run.smithi121:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:12:19.538 INFO:teuthology.orchestra.run.smithi121.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:12:19.539 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:12:19.539 INFO:teuthology.orchestra.run.smithi121.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:12:19.539 INFO:teuthology.orchestra.run.smithi121.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:12:19.539 INFO:teuthology.orchestra.run.smithi121.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:12:19.539 INFO:teuthology.orchestra.run.smithi121.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:12:19.539 INFO:teuthology.orchestra.run.smithi121.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:12:19.539 INFO:teuthology.orchestra.run.smithi121.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:12:19.539 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:12:19.539 INFO:teuthology.orchestra.run.smithi121.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:12:20.008 INFO:teuthology.orchestra.run.smithi121.stdout:Discarding blocks...Done. 2024-03-29T11:12:20.010 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi121.front.sepia.ceph.com -o noatime 2024-03-29T11:12:20.011 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-29T11:12:20.105 DEBUG:teuthology.orchestra.run.smithi121:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-29T11:12:20.136 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-29T11:12:20.206 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:12:20.206 INFO:tasks.ceph:role: osd.6 2024-03-29T11:12:20.206 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi121.front.sepia.ceph.com 2024-03-29T11:12:20.206 DEBUG:teuthology.orchestra.run.smithi121:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:12:20.280 INFO:teuthology.orchestra.run.smithi121.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:12:20.280 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:12:20.280 INFO:teuthology.orchestra.run.smithi121.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:12:20.280 INFO:teuthology.orchestra.run.smithi121.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:12:20.280 INFO:teuthology.orchestra.run.smithi121.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:12:20.281 INFO:teuthology.orchestra.run.smithi121.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:12:20.281 INFO:teuthology.orchestra.run.smithi121.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:12:20.281 INFO:teuthology.orchestra.run.smithi121.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:12:20.281 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:12:20.281 INFO:teuthology.orchestra.run.smithi121.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:12:20.781 INFO:teuthology.orchestra.run.smithi121.stdout:Discarding blocks...Done. 2024-03-29T11:12:20.783 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi121.front.sepia.ceph.com -o noatime 2024-03-29T11:12:20.783 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-29T11:12:20.934 DEBUG:teuthology.orchestra.run.smithi121:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-29T11:12:20.969 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-29T11:12:21.037 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:12:21.037 INFO:tasks.ceph:role: osd.7 2024-03-29T11:12:21.038 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi121.front.sepia.ceph.com 2024-03-29T11:12:21.038 DEBUG:teuthology.orchestra.run.smithi121:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:12:21.112 INFO:teuthology.orchestra.run.smithi121.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:12:21.113 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:12:21.113 INFO:teuthology.orchestra.run.smithi121.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:12:21.113 INFO:teuthology.orchestra.run.smithi121.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:12:21.113 INFO:teuthology.orchestra.run.smithi121.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:12:21.113 INFO:teuthology.orchestra.run.smithi121.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:12:21.113 INFO:teuthology.orchestra.run.smithi121.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:12:21.113 INFO:teuthology.orchestra.run.smithi121.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:12:21.113 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:12:21.113 INFO:teuthology.orchestra.run.smithi121.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:12:21.629 INFO:teuthology.orchestra.run.smithi121.stdout:Discarding blocks...Done. 2024-03-29T11:12:21.631 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi121.front.sepia.ceph.com -o noatime 2024-03-29T11:12:21.631 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-29T11:12:21.765 DEBUG:teuthology.orchestra.run.smithi121:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-29T11:12:21.800 DEBUG:teuthology.orchestra.run.smithi121:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:12:21.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:21.930+0000 7f3834a0d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-29T11:12:21.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:21.930+0000 7f3834a0d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-29T11:12:21.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:21.930+0000 7f3834a0d640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:12:21.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:21.930+0000 7f3834a0d640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:12:21.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:21.930+0000 7f3834a0d640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:12:21.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:21.931+0000 7f3834a0d640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-29T11:12:24.753 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-29T11:12:24.783 DEBUG:teuthology.orchestra.run.smithi121:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:12:24.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:24.894+0000 7f9ba1214640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-29T11:12:24.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:24.895+0000 7f9ba1214640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-29T11:12:24.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:24.895+0000 7f9ba1214640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:12:24.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:24.895+0000 7f9ba1214640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:12:24.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:24.895+0000 7f9ba1214640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:12:24.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:24.895+0000 7f9ba1214640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-29T11:12:27.715 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-29T11:12:27.746 DEBUG:teuthology.orchestra.run.smithi121:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:12:27.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:27.859+0000 7f34360a2640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-29T11:12:27.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:27.859+0000 7f34360a2640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-29T11:12:27.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:27.859+0000 7f34360a2640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:12:27.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:27.859+0000 7f34360a2640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:12:27.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:27.859+0000 7f34360a2640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:12:27.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:27.860+0000 7f34360a2640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-29T11:12:30.690 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-29T11:12:30.718 DEBUG:teuthology.orchestra.run.smithi121:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:12:30.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:30.826+0000 7f8716fbb640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-29T11:12:30.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:30.826+0000 7f8716fbb640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-29T11:12:30.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:30.826+0000 7f8716fbb640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:12:30.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:30.826+0000 7f8716fbb640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:12:30.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:30.826+0000 7f8716fbb640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:12:30.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:30.826+0000 7f8716fbb640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-29T11:12:33.646 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-29T11:12:33.676 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-29T11:12:33.677 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-29T11:12:33.677 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-29T11:12:33.708 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-29T11:12:33.709 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-29T11:12:33.780 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-29T11:12:33.780 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-29T11:12:33.851 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-29T11:12:33.851 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-29T11:12:33.920 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-29T11:12:33.920 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-29T11:12:33.990 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-29T11:12:33.990 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-29T11:12:34.022 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-29T11:12:34.022 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-29T11:12:34.092 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-29T11:12:34.092 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-29T11:12:34.161 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-29T11:12:34.161 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-29T11:12:34.230 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-29T11:12:34.230 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-29T11:12:34.297 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-29T11:12:34.297 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-29T11:12:34.315 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-29T11:12:34.316 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-29T11:12:34.354 INFO:tasks.ceph:Adding keys to all mons... 2024-03-29T11:12:34.354 DEBUG:teuthology.orchestra.run.smithi047:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:12:34.358 DEBUG:teuthology.orchestra.run.smithi121:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:12:34.397 INFO:teuthology.orchestra.run.smithi047.stdout:[mgr.y] 2024-03-29T11:12:34.397 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQAFogZmHm/uEhAAsE5OxY4mUs9B1tdG7oaG6g== 2024-03-29T11:12:34.397 INFO:teuthology.orchestra.run.smithi047.stdout:[osd.0] 2024-03-29T11:12:34.397 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQAGogZmG2UyMxAA+laBjt8YbR7nLVPtY/1Wcw== 2024-03-29T11:12:34.397 INFO:teuthology.orchestra.run.smithi047.stdout:[osd.1] 2024-03-29T11:12:34.397 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQAJogZm4hh0MRAAzLdgnyNKKupSOswzWSmdfA== 2024-03-29T11:12:34.397 INFO:teuthology.orchestra.run.smithi047.stdout:[osd.2] 2024-03-29T11:12:34.397 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQAMogZma73VLxAAEWFGQlf/uiQ4qZ/NQkp0rg== 2024-03-29T11:12:34.397 INFO:teuthology.orchestra.run.smithi047.stdout:[osd.3] 2024-03-29T11:12:34.397 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQAPogZmXuexLhAAwQagPxsBidTRxCx1o9Vhpg== 2024-03-29T11:12:34.398 INFO:teuthology.orchestra.run.smithi047.stdout:[mgr.x] 2024-03-29T11:12:34.398 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQAFogZmH3TIFhAAzAwISLEN5KvufVBxfuCXQQ== 2024-03-29T11:12:34.398 INFO:teuthology.orchestra.run.smithi047.stdout:[osd.4] 2024-03-29T11:12:34.398 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQAVogZmamiFNxAAoytdCu3xhZRJs5zL8kmNuA== 2024-03-29T11:12:34.398 INFO:teuthology.orchestra.run.smithi047.stdout:[osd.5] 2024-03-29T11:12:34.398 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQAYogZmFDllNRAAWDypfPIHEUsEQDvnndwJMA== 2024-03-29T11:12:34.398 INFO:teuthology.orchestra.run.smithi047.stdout:[osd.6] 2024-03-29T11:12:34.398 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQAbogZmuIxLMxAAjcvmxVSVnZqTP71TTJWwzw== 2024-03-29T11:12:34.398 INFO:teuthology.orchestra.run.smithi047.stdout:[osd.7] 2024-03-29T11:12:34.398 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQAeogZm/qdKMRAARYkQbinVbGT+4XT9nXVSPQ== 2024-03-29T11:12:34.398 INFO:teuthology.orchestra.run.smithi047.stdout:[client.0] 2024-03-29T11:12:34.398 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQAFogZmbTF3GRAAC3+Czy4684Nt/hMRB36GIw== 2024-03-29T11:12:34.398 INFO:teuthology.orchestra.run.smithi047.stdout:[client.1] 2024-03-29T11:12:34.398 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQAFogZmYzXrHBAAQKplSO3E1R1QqL8X83Hq8g== 2024-03-29T11:12:34.420 INFO:teuthology.orchestra.run.smithi121.stdout:[mgr.y] 2024-03-29T11:12:34.420 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQAFogZmHm/uEhAAsE5OxY4mUs9B1tdG7oaG6g== 2024-03-29T11:12:34.420 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.0] 2024-03-29T11:12:34.420 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQAGogZmG2UyMxAA+laBjt8YbR7nLVPtY/1Wcw== 2024-03-29T11:12:34.420 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.1] 2024-03-29T11:12:34.420 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQAJogZm4hh0MRAAzLdgnyNKKupSOswzWSmdfA== 2024-03-29T11:12:34.420 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.2] 2024-03-29T11:12:34.420 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQAMogZma73VLxAAEWFGQlf/uiQ4qZ/NQkp0rg== 2024-03-29T11:12:34.420 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.3] 2024-03-29T11:12:34.421 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQAPogZmXuexLhAAwQagPxsBidTRxCx1o9Vhpg== 2024-03-29T11:12:34.421 INFO:teuthology.orchestra.run.smithi121.stdout:[mgr.x] 2024-03-29T11:12:34.421 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQAFogZmH3TIFhAAzAwISLEN5KvufVBxfuCXQQ== 2024-03-29T11:12:34.421 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.4] 2024-03-29T11:12:34.421 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQAVogZmamiFNxAAoytdCu3xhZRJs5zL8kmNuA== 2024-03-29T11:12:34.421 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.5] 2024-03-29T11:12:34.421 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQAYogZmFDllNRAAWDypfPIHEUsEQDvnndwJMA== 2024-03-29T11:12:34.421 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.6] 2024-03-29T11:12:34.421 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQAbogZmuIxLMxAAjcvmxVSVnZqTP71TTJWwzw== 2024-03-29T11:12:34.421 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.7] 2024-03-29T11:12:34.421 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQAeogZm/qdKMRAARYkQbinVbGT+4XT9nXVSPQ== 2024-03-29T11:12:34.422 INFO:teuthology.orchestra.run.smithi121.stdout:[client.0] 2024-03-29T11:12:34.422 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQAFogZmbTF3GRAAC3+Czy4684Nt/hMRB36GIw== 2024-03-29T11:12:34.422 INFO:teuthology.orchestra.run.smithi121.stdout:[client.1] 2024-03-29T11:12:34.422 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQAFogZmYzXrHBAAQKplSO3E1R1QqL8X83Hq8g== 2024-03-29T11:12:34.423 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:12:34.442 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:12:34.515 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:12:34.517 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:12:34.571 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:12:34.573 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:12:34.625 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:12:34.627 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:12:34.679 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:12:34.681 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:12:34.733 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:12:34.735 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:12:34.786 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:12:34.788 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:12:34.840 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:12:34.842 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:12:34.894 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:12:34.896 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:12:34.948 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:12:34.950 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:12:35.001 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:12:35.004 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:12:35.059 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:12:35.061 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:12:35.115 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-29T11:12:35.116 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-29T11:12:35.142 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:12:35.241 DEBUG:teuthology.orchestra.run.smithi047:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-29T11:12:35.267 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-29T11:12:35.335 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:12:35.430 DEBUG:teuthology.orchestra.run.smithi047:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-29T11:12:35.457 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-29T11:12:35.484 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:12:35.585 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-29T11:12:35.612 DEBUG:teuthology.orchestra.run.smithi047:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:12:35.614 DEBUG:teuthology.orchestra.run.smithi121:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:12:35.666 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-29T11:12:35.667 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-29T11:12:35.667 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-29T11:12:35.669 INFO:tasks.ceph.mon.a:Started 2024-03-29T11:12:35.669 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-29T11:12:35.669 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-29T11:12:35.671 INFO:tasks.ceph.mon.c:Started 2024-03-29T11:12:35.671 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-29T11:12:35.671 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-29T11:12:35.710 INFO:tasks.ceph.mon.b:Started 2024-03-29T11:12:35.710 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-29T11:12:35.710 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-29T11:12:35.710 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-29T11:12:35.714 INFO:tasks.ceph.mgr.y:Started 2024-03-29T11:12:35.715 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-29T11:12:35.715 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-29T11:12:35.717 INFO:tasks.ceph.mgr.x:Started 2024-03-29T11:12:35.717 INFO:tasks.ceph:Setting crush tunables to default 2024-03-29T11:12:35.717 DEBUG:teuthology.orchestra.run.smithi047:> sudo ceph --cluster ceph osd crush tunables default 2024-03-29T11:12:35.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:35.841+0000 7f971176b640 10 monclient: get_monmap_and_config 2024-03-29T11:12:35.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:35.841+0000 7f971176b640 10 monclient: build_initial_monmap 2024-03-29T11:12:35.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:35.841+0000 7f971176b640 10 monclient: monmap: 2024-03-29T11:12:35.819 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-29T11:12:35.819 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:35.819 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-29T11:12:35.842498+0000 2024-03-29T11:12:35.820 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-29T11:12:35.842498+0000 2024-03-29T11:12:35.820 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:35.820 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-29T11:12:35.820 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:12:35.820 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:12:35.820 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:12:35.820 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-29T11:12:35.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:35.842+0000 7f971176b640 1 Processor -- start 2024-03-29T11:12:35.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:35.842+0000 7f971176b640 1 -- start start 2024-03-29T11:12:35.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:35.842+0000 7f971176b640 10 monclient: init 2024-03-29T11:12:35.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:35.842+0000 7f971176b640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:35.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:35.842+0000 7f971176b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:12:35.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:35.842+0000 7f971176b640 10 monclient(hunting): picked mon.noname-a con 0x7f970c09ac40 addr v1:172.21.15.47:6789/0 2024-03-29T11:12:35.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:35.842+0000 7f971176b640 10 monclient(hunting): picked mon.noname-b con 0x7f970c09c870 addr v1:172.21.15.47:6790/0 2024-03-29T11:12:35.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:35.842+0000 7f971176b640 10 monclient(hunting): picked mon.noname-c con 0x7f970c0b4360 addr v1:172.21.15.121:6789/0 2024-03-29T11:12:35.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:35.842+0000 7f971176b640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f970c09dec0 con 0x7f970c09ac40 2024-03-29T11:12:35.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:35.842+0000 7f971176b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f970c09da30 con 0x7f970c0b4360 2024-03-29T11:12:35.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:35.842+0000 7f971176b640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f970c0b39c0 con 0x7f970c09c870 2024-03-29T11:12:35.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:35.842+0000 7f971176b640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:35.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:35.842+0000 7f971176b640 10 monclient(hunting): authenticate will time out at 799.030853s 2024-03-29T11:12:35.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:35.843+0000 7f970affd640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f970c09ac40 0x7f970c0b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:48210/0 (socket says 172.21.15.47:48210) 2024-03-29T11:12:35.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:35.843+0000 7f970affd640 1 -- 172.21.15.47:0/640590643 learned_addr learned my addr 172.21.15.47:0/640590643 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:12:35.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:35.843+0000 7f9703fff640 1 --1- 172.21.15.47:0/640590643 >> v1:172.21.15.47:6790/0 conn(0x7f970c09c870 0x7f970c13f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:58802/0 (socket says 172.21.15.47:58802) 2024-03-29T11:12:38.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:38.843+0000 7f97097fa640 10 monclient(hunting): tick 2024-03-29T11:12:38.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:38.843+0000 7f97097fa640 10 monclient(hunting): _check_auth_tickets 2024-03-29T11:12:38.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:38.843+0000 7f97097fa640 1 monclient(hunting): continuing hunt 2024-03-29T11:12:38.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:38.843+0000 7f97097fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-29T11:12:38.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:38.843+0000 7f97097fa640 1 -- 172.21.15.47:0/640590643 >> v1:172.21.15.47:6790/0 conn(0x7f970c09c870 legacy=0x7f970c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:38.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:38.843+0000 7f97097fa640 1 -- 172.21.15.47:0/640590643 >> v1:172.21.15.121:6789/0 conn(0x7f970c0b4360 legacy=0x7f970c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:38.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:38.843+0000 7f97097fa640 1 -- 172.21.15.47:0/640590643 >> v1:172.21.15.47:6789/0 conn(0x7f970c09ac40 legacy=0x7f970c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:38.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:38.843+0000 7f97097fa640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:12:38.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:38.843+0000 7f97097fa640 10 monclient(hunting): picked mon.noname-b con 0x7f96ec0036e0 addr v1:172.21.15.47:6790/0 2024-03-29T11:12:38.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:38.843+0000 7f97097fa640 10 monclient(hunting): picked mon.noname-a con 0x7f96ec0072d0 addr v1:172.21.15.47:6789/0 2024-03-29T11:12:38.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:38.843+0000 7f97097fa640 10 monclient(hunting): picked mon.noname-c con 0x7f96ec00afd0 addr v1:172.21.15.121:6789/0 2024-03-29T11:12:38.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:38.843+0000 7f97097fa640 1 -- 172.21.15.47:0/640590643 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f970c0b5160 con 0x7f96ec0072d0 2024-03-29T11:12:38.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:38.843+0000 7f97097fa640 1 -- 172.21.15.47:0/640590643 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96ec00fca0 con 0x7f96ec00afd0 2024-03-29T11:12:38.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:38.843+0000 7f97097fa640 1 -- 172.21.15.47:0/640590643 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96ec010ee0 con 0x7f96ec0036e0 2024-03-29T11:12:38.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:38.843+0000 7f97097fa640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:40.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.011+0000 7f970a7fc640 1 -- 172.21.15.47:0/640590643 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3215996787 0 0) 0x7f970c0b5160 con 0x7f96ec0072d0 2024-03-29T11:12:40.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.011+0000 7f970a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:12:40.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.011+0000 7f970a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:40.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.011+0000 7f970a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:40.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.011+0000 7f970a7fc640 10 monclient(hunting): my global_id is 4101 2024-03-29T11:12:40.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.011+0000 7f970a7fc640 1 -- 172.21.15.47:0/640590643 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96f8003520 con 0x7f96ec0072d0 2024-03-29T11:12:40.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.011+0000 7f970a7fc640 1 -- 172.21.15.47:0/640590643 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1042361155 0 0) 0x7f96ec00fca0 con 0x7f96ec00afd0 2024-03-29T11:12:40.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.011+0000 7f970a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:12:40.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.011+0000 7f970a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:40.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.011+0000 7f970a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:40.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.011+0000 7f970a7fc640 10 monclient(hunting): my global_id is 4105 2024-03-29T11:12:40.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.011+0000 7f970a7fc640 1 -- 172.21.15.47:0/640590643 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f970c0b5160 con 0x7f96ec00afd0 2024-03-29T11:12:40.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.011+0000 7f970a7fc640 1 -- 172.21.15.47:0/640590643 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 382361703 0 0) 0x7f96ec010ee0 con 0x7f96ec0036e0 2024-03-29T11:12:40.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.011+0000 7f970a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:12:40.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.011+0000 7f970a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:40.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.011+0000 7f970a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:40.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.011+0000 7f970a7fc640 10 monclient(hunting): my global_id is 4106 2024-03-29T11:12:40.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.011+0000 7f970a7fc640 1 -- 172.21.15.47:0/640590643 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96ec00fca0 con 0x7f96ec0036e0 2024-03-29T11:12:40.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.011+0000 7f970a7fc640 1 -- 172.21.15.47:0/640590643 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3318367408 0 0) 0x7f970c0b5160 con 0x7f96ec00afd0 2024-03-29T11:12:40.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.011+0000 7f970a7fc640 1 -- 172.21.15.47:0/640590643 >> v1:172.21.15.47:6790/0 conn(0x7f96ec0036e0 legacy=0x7f96ec005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:40.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.011+0000 7f970a7fc640 1 -- 172.21.15.47:0/640590643 >> v1:172.21.15.47:6789/0 conn(0x7f96ec0072d0 legacy=0x7f96ec009740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:40.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.011+0000 7f970a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:12:40.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.011+0000 7f970a7fc640 1 monclient: found mon.noname-c 2024-03-29T11:12:40.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.011+0000 7f970a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:40.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.011+0000 7f970a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:12:40.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.011+0000 7f970a7fc640 1 -- 172.21.15.47:0/640590643 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96ec012140 con 0x7f96ec00afd0 2024-03-29T11:12:40.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.011+0000 7f970a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:12:40.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.011+0000 7f970a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:12:40.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.011+0000 7f970a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:40.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.011+0000 7f970a7fc640 1 -- 172.21.15.47:0/640590643 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f9704001a10 con 0x7f96ec00afd0 2024-03-29T11:12:40.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.011+0000 7f971176b640 5 monclient: authenticate success, global_id 4105 2024-03-29T11:12:40.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.011+0000 7f971176b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:40.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.012+0000 7f970a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:40.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.012+0000 7f970a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:12:40.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.012+0000 7f970a7fc640 10 monclient: dump: 2024-03-29T11:12:40.993 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-29T11:12:40.993 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:40.994 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:40.994 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:40.994 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:40.994 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-29T11:12:40.994 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:40.994 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:40.994 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:40.994 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-29T11:12:40.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.012+0000 7f971176b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:40.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.012+0000 7f970a7fc640 1 -- 172.21.15.47:0/640590643 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9704001be0 con 0x7f96ec00afd0 2024-03-29T11:12:40.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.012+0000 7f970a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:40.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.012+0000 7f970a7fc640 1 -- 172.21.15.47:0/640590643 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f9704001fb0 con 0x7f96ec00afd0 2024-03-29T11:12:40.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.012+0000 7f970a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:40.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.012+0000 7f970a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:40.995 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.012+0000 7f970a7fc640 10 monclient: dump: 2024-03-29T11:12:40.995 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-29T11:12:40.995 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:40.995 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:40.995 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:40.995 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:40.995 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-29T11:12:40.995 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:40.995 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:40.995 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:40.995 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-29T11:12:40.995 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.012+0000 7f971176b640 10 monclient: get_monmap_and_config success 2024-03-29T11:12:40.995 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.012+0000 7f971176b640 4 set_mon_vals no callback set 2024-03-29T11:12:40.995 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.013+0000 7f971176b640 10 monclient: shutdown 2024-03-29T11:12:40.995 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.013+0000 7f971176b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:40.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.013+0000 7f971176b640 1 -- 172.21.15.47:0/640590643 >> v1:172.21.15.121:6789/0 conn(0x7f96ec00afd0 legacy=0x7f96ec00d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:40.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.013+0000 7f971176b640 1 -- 172.21.15.47:0/640590643 shutdown_connections 2024-03-29T11:12:40.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.013+0000 7f971176b640 1 -- 172.21.15.47:0/640590643 >> 172.21.15.47:0/640590643 conn(0x7f970c066b00 msgr2=0x7f970c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:40.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.013+0000 7f971176b640 1 -- 172.21.15.47:0/640590643 shutdown_connections 2024-03-29T11:12:40.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.013+0000 7f971176b640 1 -- 172.21.15.47:0/640590643 wait complete. 2024-03-29T11:12:41.000 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.014+0000 7f971176b640 10 monclient: build_initial_monmap 2024-03-29T11:12:41.000 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.014+0000 7f971176b640 10 monclient: monmap: 2024-03-29T11:12:41.000 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-29T11:12:41.000 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:12:41.001 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-29T11:12:41.001 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-29T11:12:41.001 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:41.001 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-29T11:12:41.001 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:12:41.001 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:12:41.001 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:12:41.001 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-29T11:12:41.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.014+0000 7f971176b640 1 librados: starting msgr at 2024-03-29T11:12:41.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.014+0000 7f971176b640 1 librados: starting objecter 2024-03-29T11:12:41.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.014+0000 7f971176b640 1 Processor -- start 2024-03-29T11:12:41.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.014+0000 7f971176b640 1 -- start start 2024-03-29T11:12:41.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.014+0000 7f971176b640 1 librados: setting wanted keys 2024-03-29T11:12:41.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.014+0000 7f971176b640 1 librados: calling monclient init 2024-03-29T11:12:41.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.014+0000 7f971176b640 10 monclient: init 2024-03-29T11:12:41.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.015+0000 7f971176b640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:41.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.015+0000 7f971176b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:12:41.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.015+0000 7f971176b640 10 monclient(hunting): picked mon.noname-a con 0x7f96ec0036e0 addr v1:172.21.15.47:6789/0 2024-03-29T11:12:41.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.015+0000 7f971176b640 10 monclient(hunting): picked mon.noname-c con 0x7f96ec0072d0 addr v1:172.21.15.47:6790/0 2024-03-29T11:12:41.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.015+0000 7f971176b640 10 monclient(hunting): picked mon.noname-b con 0x7f96ec00afd0 addr v1:172.21.15.121:6789/0 2024-03-29T11:12:41.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.015+0000 7f971176b640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f970c145080 con 0x7f96ec0036e0 2024-03-29T11:12:41.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.015+0000 7f971176b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f970c1452f0 con 0x7f96ec00afd0 2024-03-29T11:12:41.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.015+0000 7f971176b640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f970c1437d0 con 0x7f96ec0072d0 2024-03-29T11:12:41.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.015+0000 7f971176b640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:41.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.015+0000 7f9703fff640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f96ec0072d0 0x7f970c138e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:54790/0 (socket says 172.21.15.47:54790) 2024-03-29T11:12:41.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.015+0000 7f9703fff640 1 -- 172.21.15.47:0/1332865791 learned_addr learned my addr 172.21.15.47:0/1332865791 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:12:41.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.015+0000 7f971176b640 10 monclient(hunting): authenticate will time out at 804.203887s 2024-03-29T11:12:41.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.016+0000 7f97037fe640 10 client.?.objecter ms_handle_connect 0x7f96ec0072d0 2024-03-29T11:12:41.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.016+0000 7f97037fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:41.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.016+0000 7f97037fe640 10 client.?.objecter ms_handle_connect 0x7f96ec00afd0 2024-03-29T11:12:41.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.016+0000 7f97037fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:41.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.016+0000 7f97037fe640 10 client.?.objecter ms_handle_connect 0x7f96ec0036e0 2024-03-29T11:12:41.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.016+0000 7f97037fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:41.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.016+0000 7f97037fe640 1 -- 172.21.15.47:0/1332865791 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3861532141 0 0) 0x7f970c145080 con 0x7f96ec0036e0 2024-03-29T11:12:41.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.016+0000 7f97037fe640 10 monclient(hunting): _negotiate 2024-03-29T11:12:41.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.016+0000 7f97037fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:41.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.016+0000 7f97037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:41.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.016+0000 7f97037fe640 10 monclient(hunting): my global_id is 4107 2024-03-29T11:12:41.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.016+0000 7f97037fe640 1 -- 172.21.15.47:0/1332865791 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96ec00a630 con 0x7f96ec0036e0 2024-03-29T11:12:41.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.016+0000 7f97037fe640 1 -- 172.21.15.47:0/1332865791 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1617926234 0 0) 0x7f96ec00a630 con 0x7f96ec0036e0 2024-03-29T11:12:41.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.016+0000 7f97037fe640 1 -- 172.21.15.47:0/1332865791 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f970c145080 con 0x7f96ec0036e0 2024-03-29T11:12:41.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.016+0000 7f97037fe640 1 -- 172.21.15.47:0/1332865791 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f96fc001750 con 0x7f96ec0036e0 2024-03-29T11:12:41.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.016+0000 7f97037fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:41.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.016+0000 7f97037fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:12:41.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.016+0000 7f97037fe640 10 monclient(hunting): dump: 2024-03-29T11:12:41.004 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-29T11:12:41.004 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:41.004 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:41.004 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:41.004 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:41.004 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-29T11:12:41.004 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:41.004 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:41.005 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:41.005 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-29T11:12:41.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.016+0000 7f97037fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:12:41.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.017+0000 7f97037fe640 1 -- 172.21.15.47:0/1332865791 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3981160696 0 0) 0x7f970c145080 con 0x7f96ec0036e0 2024-03-29T11:12:41.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.017+0000 7f97037fe640 1 -- 172.21.15.47:0/1332865791 >> v1:172.21.15.47:6790/0 conn(0x7f96ec0072d0 legacy=0x7f970c138e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:41.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.017+0000 7f97037fe640 1 -- 172.21.15.47:0/1332865791 >> v1:172.21.15.121:6789/0 conn(0x7f96ec00afd0 legacy=0x7f970c1e2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:41.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.017+0000 7f97037fe640 10 monclient: _finish_hunting 0 2024-03-29T11:12:41.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.017+0000 7f97037fe640 1 monclient: found mon.a 2024-03-29T11:12:41.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.017+0000 7f97037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:41.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.017+0000 7f97037fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:12:41.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.017+0000 7f97037fe640 1 -- 172.21.15.47:0/1332865791 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f970c1434a0 con 0x7f96ec0036e0 2024-03-29T11:12:41.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.017+0000 7f97037fe640 10 monclient: _finish_auth 0 2024-03-29T11:12:41.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.017+0000 7f97037fe640 10 monclient: _check_auth_tickets 2024-03-29T11:12:41.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.017+0000 7f97037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:41.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.017+0000 7f971176b640 5 monclient: authenticate success, global_id 4107 2024-03-29T11:12:41.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.017+0000 7f971176b640 10 monclient: _renew_subs 2024-03-29T11:12:41.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.017+0000 7f971176b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:12:41.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.017+0000 7f971176b640 1 -- 172.21.15.47:0/1332865791 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f970c145e20 con 0x7f96ec0036e0 2024-03-29T11:12:41.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.017+0000 7f971176b640 10 client.4107.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:12:41.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.017+0000 7f971176b640 10 monclient: _renew_subs 2024-03-29T11:12:41.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.017+0000 7f971176b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:12:41.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.017+0000 7f971176b640 1 -- 172.21.15.47:0/1332865791 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f970c139d60 con 0x7f96ec0036e0 2024-03-29T11:12:41.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.017+0000 7f971176b640 1 librados: init done 2024-03-29T11:12:41.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.017+0000 7f97037fe640 1 -- 172.21.15.47:0/1332865791 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f96fc001ce0 con 0x7f96ec0036e0 2024-03-29T11:12:41.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.017+0000 7f97037fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:41.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.017+0000 7f97037fe640 1 -- 172.21.15.47:0/1332865791 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f96fc004980 con 0x7f96ec0036e0 2024-03-29T11:12:41.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.017+0000 7f97037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:41.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.017+0000 7f97037fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:12:41.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.017+0000 7f97037fe640 10 monclient: dump: 2024-03-29T11:12:41.007 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-29T11:12:41.007 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:41.007 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:41.007 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:41.007 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:41.007 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-29T11:12:41.007 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:41.007 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:41.007 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:41.007 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-29T11:12:41.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.017+0000 7f97037fe640 1 -- 172.21.15.47:0/1332865791 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f96fc004eb0 con 0x7f96ec0036e0 2024-03-29T11:12:41.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.017+0000 7f9709ffb640 4 set_mon_vals no callback set 2024-03-29T11:12:41.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.017+0000 7f97037fe640 1 -- 172.21.15.47:0/1332865791 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 2803480047 0 0) 0x7f96fc006430 con 0x7f96ec0036e0 2024-03-29T11:12:41.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.017+0000 7f97037fe640 10 client.4107.objecter ms_dispatch 0x7f970c000b90 osd_map(1..1 src has 1..1) v4 2024-03-29T11:12:41.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.017+0000 7f97037fe640 3 client.4107.objecter handle_osd_map got epochs [1,1] > 0 2024-03-29T11:12:41.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.017+0000 7f97037fe640 3 client.4107.objecter handle_osd_map decoding full epoch 1 2024-03-29T11:12:41.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.017+0000 7f97037fe640 20 client.4107.objecter dump_active .. 0 homeless 2024-03-29T11:12:41.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.018+0000 7f971176b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:41.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.018+0000 7f971176b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:41.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.018+0000 7f971176b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:12:41.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.018+0000 7f971176b640 1 -- 172.21.15.47:0/1332865791 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96f0005080 con 0x7f96ec0036e0 2024-03-29T11:12:41.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.020+0000 7f97037fe640 1 -- 172.21.15.47:0/1332865791 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f96fc005440 con 0x7f96ec0036e0 2024-03-29T11:12:41.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.020+0000 7f97037fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:41.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.020+0000 7f97037fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:12:41.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.093+0000 7f971176b640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:12:41.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.093+0000 7f971176b640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:12:41.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.093+0000 7f971176b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:12:41.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.093+0000 7f971176b640 1 -- 172.21.15.47:0/1332865791 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f96f0005330 con 0x7f96ec0036e0 2024-03-29T11:12:41.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.095+0000 7f97037fe640 1 -- 172.21.15.47:0/1332865791 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f96fc005740 con 0x7f96ec0036e0 2024-03-29T11:12:41.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.095+0000 7f97037fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:12:41.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.095+0000 7f97037fe640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-29T11:12:41.073 INFO:teuthology.orchestra.run.smithi047.stderr:adjusted tunables profile to default 2024-03-29T11:12:41.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.096+0000 7f971176b640 10 librados: watch_flush enter 2024-03-29T11:12:41.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.096+0000 7f971176b640 10 librados: watch_flush exit 2024-03-29T11:12:41.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.096+0000 7f971176b640 20 client.4107.objecter shutdown clearing up homeless session... 2024-03-29T11:12:41.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.096+0000 7f971176b640 10 client.4107.objecter successfully canceled tick 2024-03-29T11:12:41.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.096+0000 7f971176b640 10 monclient: shutdown 2024-03-29T11:12:41.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.096+0000 7f971176b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:41.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.096+0000 7f971176b640 1 -- 172.21.15.47:0/1332865791 >> v1:172.21.15.47:6789/0 conn(0x7f96ec0036e0 legacy=0x7f970c145710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:41.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.096+0000 7f971176b640 1 -- 172.21.15.47:0/1332865791 shutdown_connections 2024-03-29T11:12:41.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.096+0000 7f971176b640 1 -- 172.21.15.47:0/1332865791 >> 172.21.15.47:0/1332865791 conn(0x7f970c066b00 msgr2=0x7f970c069bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:41.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.096+0000 7f971176b640 1 -- 172.21.15.47:0/1332865791 shutdown_connections 2024-03-29T11:12:41.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.096+0000 7f971176b640 1 -- 172.21.15.47:0/1332865791 wait complete. 2024-03-29T11:12:41.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:41.096+0000 7f971176b640 1 librados: shutdown 2024-03-29T11:12:41.085 INFO:tasks.ceph:check_enable_crimson: False 2024-03-29T11:12:41.085 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-29T11:12:41.086 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-29T11:12:41.086 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-29T11:12:41.113 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-29T11:12:41.113 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-29T11:12:41.180 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-29T11:12:41.180 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-29T11:12:41.247 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-29T11:12:41.247 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-29T11:12:41.315 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-29T11:12:41.315 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-29T11:12:41.342 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-29T11:12:41.342 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-29T11:12:41.409 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-29T11:12:41.409 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-29T11:12:41.476 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-29T11:12:41.476 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-29T11:12:41.495 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-29T11:12:41.517+0000 7fcab480f040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:12:41.518 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-29T11:12:41.540+0000 7f3465a3c040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:12:41.543 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new 19bcf217-b930-4b2c-84c1-56b8b18c9805 0 2024-03-29T11:12:41.550 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-29T11:12:41.573+0000 7fcab480f040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:12:41.574 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-29T11:12:41.596+0000 7f3465a3c040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:12:41.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.695+0000 7fddde055640 10 monclient: get_monmap_and_config 2024-03-29T11:12:41.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.695+0000 7fddde055640 10 monclient: build_initial_monmap 2024-03-29T11:12:41.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.695+0000 7fddde055640 10 monclient: monmap: 2024-03-29T11:12:41.691 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-03-29T11:12:41.691 INFO:teuthology.orchestra.run.smithi121.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:41.691 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:12:41.696528+0000 2024-03-29T11:12:41.692 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:12:41.696528+0000 2024-03-29T11:12:41.692 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:41.692 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:41.692 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:12:41.692 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:12:41.692 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:12:41.692 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:41.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.696+0000 7fddde055640 1 Processor -- start 2024-03-29T11:12:41.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.696+0000 7fddde055640 1 -- start start 2024-03-29T11:12:41.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.696+0000 7fddde055640 10 monclient: init 2024-03-29T11:12:41.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.696+0000 7fddde055640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:41.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.696+0000 7fddde055640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:12:41.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.696+0000 7fddde055640 10 monclient(hunting): picked mon.noname-a con 0x7fddd80b8240 addr v1:172.21.15.47:6789/0 2024-03-29T11:12:41.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.696+0000 7fddde055640 10 monclient(hunting): picked mon.noname-c con 0x7fddd80b1f60 addr v1:172.21.15.121:6789/0 2024-03-29T11:12:41.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.696+0000 7fddde055640 10 monclient(hunting): picked mon.noname-b con 0x7fddd809c8c0 addr v1:172.21.15.47:6790/0 2024-03-29T11:12:41.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.696+0000 7fddde055640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddd809bd80 con 0x7fddd80b8240 2024-03-29T11:12:41.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.697+0000 7fddde055640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddd809b960 con 0x7fddd80b1f60 2024-03-29T11:12:41.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.697+0000 7fddde055640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddd80b2ec0 con 0x7fddd809c8c0 2024-03-29T11:12:41.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.697+0000 7fddde055640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:41.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.697+0000 7fddde055640 10 monclient(hunting): authenticate will time out at 815.863135s 2024-03-29T11:12:41.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.697+0000 7fddd6ffd640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fddd80b1f60 0x7fddd809c1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:54612/0 (socket says 172.21.15.121:54612) 2024-03-29T11:12:41.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.697+0000 7fddd6ffd640 1 -- 172.21.15.121:0/3456195913 learned_addr learned my addr 172.21.15.121:0/3456195913 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-03-29T11:12:41.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.697+0000 7fddd67fc640 1 -- 172.21.15.121:0/3456195913 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 500642284 0 0) 0x7fddd809b960 con 0x7fddd80b1f60 2024-03-29T11:12:41.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.697+0000 7fddd67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:12:41.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.697+0000 7fddd67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:41.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.697+0000 7fddd67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:41.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.697+0000 7fddd67fc640 10 monclient(hunting): my global_id is 4108 2024-03-29T11:12:41.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.697+0000 7fddd67fc640 1 -- 172.21.15.121:0/3456195913 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddc0003520 con 0x7fddd80b1f60 2024-03-29T11:12:41.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.697+0000 7fddd67fc640 1 -- 172.21.15.121:0/3456195913 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2102137000 0 0) 0x7fddd80b2ec0 con 0x7fddd809c8c0 2024-03-29T11:12:41.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.697+0000 7fddd67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:12:41.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.697+0000 7fddd67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:41.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.697+0000 7fddd67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:41.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.697+0000 7fddd67fc640 10 monclient(hunting): my global_id is 4112 2024-03-29T11:12:41.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.697+0000 7fddd67fc640 1 -- 172.21.15.121:0/3456195913 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddd809b960 con 0x7fddd809c8c0 2024-03-29T11:12:41.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.697+0000 7fddd67fc640 1 -- 172.21.15.121:0/3456195913 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1832560692 0 0) 0x7fddd809bd80 con 0x7fddd80b8240 2024-03-29T11:12:41.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.697+0000 7fddd67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:12:41.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.697+0000 7fddd67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:41.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.697+0000 7fddd67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:41.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.697+0000 7fddd67fc640 10 monclient(hunting): my global_id is 4113 2024-03-29T11:12:41.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.697+0000 7fddd67fc640 1 -- 172.21.15.121:0/3456195913 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddd80b2ec0 con 0x7fddd80b8240 2024-03-29T11:12:41.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.697+0000 7fddd67fc640 1 -- 172.21.15.121:0/3456195913 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4245838318 0 0) 0x7fddc0003520 con 0x7fddd80b1f60 2024-03-29T11:12:41.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.697+0000 7fddd67fc640 1 -- 172.21.15.121:0/3456195913 >> v1:172.21.15.47:6790/0 conn(0x7fddd809c8c0 legacy=0x7fddd814d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:41.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.697+0000 7fddd67fc640 1 -- 172.21.15.121:0/3456195913 >> v1:172.21.15.47:6789/0 conn(0x7fddd80b8240 legacy=0x7fddd813f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:41.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.697+0000 7fddd67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:12:41.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.697+0000 7fddd67fc640 1 monclient: found mon.noname-c 2024-03-29T11:12:41.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.697+0000 7fddd67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:41.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.697+0000 7fddd67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:12:41.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.697+0000 7fddd67fc640 1 -- 172.21.15.121:0/3456195913 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddd813fb60 con 0x7fddd80b1f60 2024-03-29T11:12:41.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.697+0000 7fddd67fc640 10 monclient: _finish_auth 0 2024-03-29T11:12:41.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.697+0000 7fddd67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:12:41.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.697+0000 7fddd67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:41.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.697+0000 7fddde055640 5 monclient: authenticate success, global_id 4108 2024-03-29T11:12:41.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.697+0000 7fddde055640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:41.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.697+0000 7fddd67fc640 1 -- 172.21.15.121:0/3456195913 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fddcc001690 con 0x7fddd80b1f60 2024-03-29T11:12:41.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.697+0000 7fddd67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:41.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.697+0000 7fddd67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:12:41.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.697+0000 7fddd67fc640 10 monclient: dump: 2024-03-29T11:12:41.698 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:12:41.698 INFO:teuthology.orchestra.run.smithi121.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:41.699 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:41.699 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:41.699 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:41.699 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:41.699 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:41.699 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:41.699 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:41.699 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:41.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.697+0000 7fddde055640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:41.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.697+0000 7fddd67fc640 1 -- 172.21.15.121:0/3456195913 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fddcc0017f0 con 0x7fddd80b1f60 2024-03-29T11:12:41.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.697+0000 7fddd67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:41.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.697+0000 7fddd67fc640 1 -- 172.21.15.121:0/3456195913 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fddcc001c00 con 0x7fddd80b1f60 2024-03-29T11:12:41.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.697+0000 7fddd67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:41.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.697+0000 7fddd67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:41.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.697+0000 7fddd67fc640 10 monclient: dump: 2024-03-29T11:12:41.701 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:12:41.701 INFO:teuthology.orchestra.run.smithi121.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:41.701 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:41.701 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:41.701 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:41.702 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:41.702 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:41.702 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:41.702 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:41.702 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:41.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.697+0000 7fddde055640 10 monclient: get_monmap_and_config success 2024-03-29T11:12:41.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.697+0000 7fddde055640 4 set_mon_vals no callback set 2024-03-29T11:12:41.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.698+0000 7fddde055640 10 monclient: shutdown 2024-03-29T11:12:41.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.698+0000 7fddde055640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:41.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.698+0000 7fddde055640 1 -- 172.21.15.121:0/3456195913 >> v1:172.21.15.121:6789/0 conn(0x7fddd80b1f60 legacy=0x7fddd809c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:41.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.698+0000 7fddde055640 1 -- 172.21.15.121:0/3456195913 shutdown_connections 2024-03-29T11:12:41.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.698+0000 7fddde055640 1 -- 172.21.15.121:0/3456195913 >> 172.21.15.121:0/3456195913 conn(0x7fddd8066b00 msgr2=0x7fddd8066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:41.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.698+0000 7fddde055640 1 -- 172.21.15.121:0/3456195913 shutdown_connections 2024-03-29T11:12:41.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.698+0000 7fddde055640 1 -- 172.21.15.121:0/3456195913 wait complete. 2024-03-29T11:12:41.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.699+0000 7fddde055640 10 monclient: build_initial_monmap 2024-03-29T11:12:41.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.699+0000 7fddde055640 10 monclient: monmap: 2024-03-29T11:12:41.704 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-03-29T11:12:41.704 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:12:41.704 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-03-29T11:12:41.704 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-03-29T11:12:41.704 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:41.704 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:41.705 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:12:41.705 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:12:41.705 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:12:41.705 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:41.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.699+0000 7fddde055640 1 librados: starting msgr at 2024-03-29T11:12:41.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.699+0000 7fddde055640 1 librados: starting objecter 2024-03-29T11:12:41.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.699+0000 7fddde055640 1 Processor -- start 2024-03-29T11:12:41.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.699+0000 7fddde055640 1 -- start start 2024-03-29T11:12:41.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.699+0000 7fddde055640 1 librados: setting wanted keys 2024-03-29T11:12:41.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.699+0000 7fddde055640 1 librados: calling monclient init 2024-03-29T11:12:41.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.699+0000 7fddde055640 10 monclient: init 2024-03-29T11:12:41.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddde055640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:41.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddde055640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:12:41.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddde055640 10 monclient(hunting): picked mon.noname-c con 0x7fddd809c8c0 addr v1:172.21.15.47:6790/0 2024-03-29T11:12:41.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddde055640 10 monclient(hunting): picked mon.noname-a con 0x7fddd80b8240 addr v1:172.21.15.47:6789/0 2024-03-29T11:12:41.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddde055640 10 monclient(hunting): picked mon.noname-b con 0x7fddd8135110 addr v1:172.21.15.121:6789/0 2024-03-29T11:12:41.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddde055640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddd80ad5a0 con 0x7fddd80b8240 2024-03-29T11:12:41.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddde055640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddd81d7710 con 0x7fddd8135110 2024-03-29T11:12:41.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddde055640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddd8134ee0 con 0x7fddd809c8c0 2024-03-29T11:12:41.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddde055640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:41.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddde055640 10 monclient(hunting): authenticate will time out at 815.866257s 2024-03-29T11:12:41.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddd7fff640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fddd8135110 0x7fddd81d46c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:54624/0 (socket says 172.21.15.121:54624) 2024-03-29T11:12:41.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddd7fff640 1 -- 172.21.15.121:0/1788955929 learned_addr learned my addr 172.21.15.121:0/1788955929 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-03-29T11:12:41.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddbffff640 10 client.?.objecter ms_handle_connect 0x7fddd8135110 2024-03-29T11:12:41.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddbffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:41.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddbffff640 10 client.?.objecter ms_handle_connect 0x7fddd809c8c0 2024-03-29T11:12:41.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddbffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:41.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddbffff640 10 client.?.objecter ms_handle_connect 0x7fddd80b8240 2024-03-29T11:12:41.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddbffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:41.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddbffff640 1 -- 172.21.15.121:0/1788955929 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 585364725 0 0) 0x7fddd81d7710 con 0x7fddd8135110 2024-03-29T11:12:41.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddbffff640 10 monclient(hunting): _negotiate 2024-03-29T11:12:41.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddbffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:41.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddbffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:41.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddbffff640 10 monclient(hunting): my global_id is 4111 2024-03-29T11:12:41.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddbffff640 1 -- 172.21.15.121:0/1788955929 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddc4004eb0 con 0x7fddd8135110 2024-03-29T11:12:41.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddbffff640 1 -- 172.21.15.121:0/1788955929 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3755583034 0 0) 0x7fddd8134ee0 con 0x7fddd809c8c0 2024-03-29T11:12:41.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddbffff640 10 monclient(hunting): _negotiate 2024-03-29T11:12:41.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddbffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:41.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddbffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:41.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddbffff640 10 monclient(hunting): my global_id is 4115 2024-03-29T11:12:41.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddbffff640 1 -- 172.21.15.121:0/1788955929 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddd81d7710 con 0x7fddd809c8c0 2024-03-29T11:12:41.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddbffff640 1 -- 172.21.15.121:0/1788955929 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4215852577 0 0) 0x7fddd80ad5a0 con 0x7fddd80b8240 2024-03-29T11:12:41.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddbffff640 10 monclient(hunting): _negotiate 2024-03-29T11:12:41.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddbffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:41.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddbffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:41.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddbffff640 10 monclient(hunting): my global_id is 4116 2024-03-29T11:12:41.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddbffff640 1 -- 172.21.15.121:0/1788955929 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddd8134ee0 con 0x7fddd80b8240 2024-03-29T11:12:41.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddbffff640 1 -- 172.21.15.121:0/1788955929 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 329137793 0 0) 0x7fddc4004eb0 con 0x7fddd8135110 2024-03-29T11:12:41.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddbffff640 1 -- 172.21.15.121:0/1788955929 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fddd80ad5a0 con 0x7fddd8135110 2024-03-29T11:12:41.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddbffff640 1 -- 172.21.15.121:0/1788955929 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fddd0001c90 con 0x7fddd8135110 2024-03-29T11:12:41.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddbffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:41.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddbffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:12:41.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddbffff640 10 monclient(hunting): dump: 2024-03-29T11:12:41.710 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:12:41.710 INFO:teuthology.orchestra.run.smithi121.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:41.710 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:41.710 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:41.710 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:41.710 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:41.710 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:41.710 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:41.711 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:41.711 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:41.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddbffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:12:41.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddbffff640 1 -- 172.21.15.121:0/1788955929 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1757917246 0 0) 0x7fddd81d7710 con 0x7fddd809c8c0 2024-03-29T11:12:41.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddbffff640 1 -- 172.21.15.121:0/1788955929 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fddc4004eb0 con 0x7fddd809c8c0 2024-03-29T11:12:41.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddbffff640 1 -- 172.21.15.121:0/1788955929 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fddc8001870 con 0x7fddd809c8c0 2024-03-29T11:12:41.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddbffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:41.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddbffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:12:41.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddbffff640 10 monclient(hunting): dump: 2024-03-29T11:12:41.711 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:12:41.711 INFO:teuthology.orchestra.run.smithi121.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:41.712 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:41.712 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:41.712 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:41.712 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:41.712 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:41.712 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:41.712 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:41.712 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:41.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddbffff640 1 -- 172.21.15.121:0/1788955929 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1124648916 0 0) 0x7fddd8134ee0 con 0x7fddd80b8240 2024-03-29T11:12:41.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddbffff640 1 -- 172.21.15.121:0/1788955929 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fddd81d7710 con 0x7fddd80b8240 2024-03-29T11:12:41.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddbffff640 1 -- 172.21.15.121:0/1788955929 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fddcc001960 con 0x7fddd80b8240 2024-03-29T11:12:41.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddbffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:41.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddbffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:12:41.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddbffff640 10 monclient(hunting): dump: 2024-03-29T11:12:41.713 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:12:41.713 INFO:teuthology.orchestra.run.smithi121.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:41.713 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:41.713 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:41.713 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:41.713 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:41.713 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:41.713 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:41.713 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:41.713 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:41.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddbffff640 1 -- 172.21.15.121:0/1788955929 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3568495895 0 0) 0x7fddd80ad5a0 con 0x7fddd8135110 2024-03-29T11:12:41.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddbffff640 1 -- 172.21.15.121:0/1788955929 >> v1:172.21.15.47:6790/0 conn(0x7fddd809c8c0 legacy=0x7fddd81347d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:41.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddbffff640 1 -- 172.21.15.121:0/1788955929 >> v1:172.21.15.47:6789/0 conn(0x7fddd80b8240 legacy=0x7fddd81d7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:41.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddbffff640 10 monclient: _finish_hunting 0 2024-03-29T11:12:41.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddbffff640 1 monclient: found mon.b 2024-03-29T11:12:41.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddbffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:41.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddbffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:12:41.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddbffff640 1 -- 172.21.15.121:0/1788955929 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddd81d82b0 con 0x7fddd8135110 2024-03-29T11:12:41.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddbffff640 10 monclient: _finish_auth 0 2024-03-29T11:12:41.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddbffff640 10 monclient: _check_auth_tickets 2024-03-29T11:12:41.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddbffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:41.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddde055640 5 monclient: authenticate success, global_id 4111 2024-03-29T11:12:41.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddde055640 10 monclient: _renew_subs 2024-03-29T11:12:41.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddde055640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:12:41.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddde055640 1 -- 172.21.15.121:0/1788955929 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddd81d73a0 con 0x7fddd8135110 2024-03-29T11:12:41.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddde055640 10 client.4111.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:12:41.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddde055640 10 monclient: _renew_subs 2024-03-29T11:12:41.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddde055640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:12:41.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddde055640 1 -- 172.21.15.121:0/1788955929 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fddd81d4fb0 con 0x7fddd8135110 2024-03-29T11:12:41.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.700+0000 7fddde055640 1 librados: init done 2024-03-29T11:12:41.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.701+0000 7fddbffff640 1 -- 172.21.15.121:0/1788955929 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fddd0003970 con 0x7fddd8135110 2024-03-29T11:12:41.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.701+0000 7fddbffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:41.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.701+0000 7fddbffff640 1 -- 172.21.15.121:0/1788955929 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fddd0003cc0 con 0x7fddd8135110 2024-03-29T11:12:41.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.701+0000 7fddbffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:41.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.701+0000 7fddbffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:41.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.701+0000 7fddd5ffb640 4 set_mon_vals no callback set 2024-03-29T11:12:41.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.701+0000 7fddbffff640 10 monclient: dump: 2024-03-29T11:12:41.716 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:12:41.716 INFO:teuthology.orchestra.run.smithi121.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:41.716 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:41.716 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:41.716 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:41.716 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:41.716 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:41.716 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:41.716 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:41.716 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:41.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.701+0000 7fddbffff640 1 -- 172.21.15.121:0/1788955929 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fddd0004270 con 0x7fddd8135110 2024-03-29T11:12:41.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.701+0000 7fddbffff640 1 -- 172.21.15.121:0/1788955929 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 3334337145 0 0) 0x7fddd00068c0 con 0x7fddd8135110 2024-03-29T11:12:41.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.701+0000 7fddbffff640 10 client.4111.objecter ms_dispatch 0x7fddd8000b90 osd_map(2..2 src has 1..2) v4 2024-03-29T11:12:41.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.701+0000 7fddbffff640 3 client.4111.objecter handle_osd_map got epochs [2,2] > 0 2024-03-29T11:12:41.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.701+0000 7fddbffff640 3 client.4111.objecter handle_osd_map decoding full epoch 2 2024-03-29T11:12:41.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.701+0000 7fddbffff640 20 client.4111.objecter dump_active .. 0 homeless 2024-03-29T11:12:41.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.701+0000 7fddde055640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:41.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.701+0000 7fddde055640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:41.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.701+0000 7fddde055640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:12:41.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.701+0000 7fddde055640 1 -- 172.21.15.121:0/1788955929 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddb8005080 con 0x7fddd8135110 2024-03-29T11:12:41.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.703+0000 7fddbffff640 1 -- 172.21.15.121:0/1788955929 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fddd0004700 con 0x7fddd8135110 2024-03-29T11:12:41.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.703+0000 7fddbffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:41.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.703+0000 7fddbffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:12:41.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.776+0000 7fddde055640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "19bcf217-b930-4b2c-84c1-56b8b18c9805", "id": 0}] 2024-03-29T11:12:41.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.776+0000 7fddde055640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "19bcf217-b930-4b2c-84c1-56b8b18c9805", "id": 0}] 2024-03-29T11:12:41.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.776+0000 7fddde055640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:12:41.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.776+0000 7fddde055640 1 -- 172.21.15.121:0/1788955929 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd new", "uuid": "19bcf217-b930-4b2c-84c1-56b8b18c9805", "id": 0} v 0) v1 -- 0x7fddb80058b0 con 0x7fddd8135110 2024-03-29T11:12:41.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.781+0000 7fddbffff640 1 -- 172.21.15.121:0/1788955929 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "19bcf217-b930-4b2c-84c1-56b8b18c9805", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 1044952733 0 732707515) 0x7fddd0006ad0 con 0x7fddd8135110 2024-03-29T11:12:41.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.781+0000 7fddbffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "19bcf217-b930-4b2c-84c1-56b8b18c9805", "id": 0}] 2024-03-29T11:12:41.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.781+0000 7fddbffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:12:41.759 INFO:teuthology.orchestra.run.smithi121.stdout:0 2024-03-29T11:12:41.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.782+0000 7fddde055640 10 librados: watch_flush enter 2024-03-29T11:12:41.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.782+0000 7fddde055640 10 librados: watch_flush exit 2024-03-29T11:12:41.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.782+0000 7fddde055640 20 client.4111.objecter shutdown clearing up homeless session... 2024-03-29T11:12:41.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.782+0000 7fddde055640 10 client.4111.objecter successfully canceled tick 2024-03-29T11:12:41.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.782+0000 7fddde055640 10 monclient: shutdown 2024-03-29T11:12:41.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.782+0000 7fddde055640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:41.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.782+0000 7fddde055640 1 -- 172.21.15.121:0/1788955929 >> v1:172.21.15.121:6789/0 conn(0x7fddd8135110 legacy=0x7fddd81d46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:41.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.782+0000 7fddde055640 1 -- 172.21.15.121:0/1788955929 shutdown_connections 2024-03-29T11:12:41.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.782+0000 7fddde055640 1 -- 172.21.15.121:0/1788955929 >> 172.21.15.121:0/1788955929 conn(0x7fddd8066b00 msgr2=0x7fddd806b230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:41.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.782+0000 7fddde055640 1 -- 172.21.15.121:0/1788955929 shutdown_connections 2024-03-29T11:12:41.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.782+0000 7fddde055640 1 -- 172.21.15.121:0/1788955929 wait complete. 2024-03-29T11:12:41.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.782+0000 7fddde055640 1 librados: shutdown 2024-03-29T11:12:41.770 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new bdab7243-0401-4284-ac2d-2802e712c07b 1 2024-03-29T11:12:41.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.891+0000 7f40673d6640 10 monclient: get_monmap_and_config 2024-03-29T11:12:41.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.891+0000 7f40673d6640 10 monclient: build_initial_monmap 2024-03-29T11:12:41.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.891+0000 7f40673d6640 10 monclient: monmap: 2024-03-29T11:12:41.870 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-03-29T11:12:41.870 INFO:teuthology.orchestra.run.smithi121.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:41.870 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:12:41.892791+0000 2024-03-29T11:12:41.870 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:12:41.892791+0000 2024-03-29T11:12:41.870 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:41.870 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:41.871 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:12:41.871 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:12:41.871 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:12:41.871 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:41.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.892+0000 7f40673d6640 1 Processor -- start 2024-03-29T11:12:41.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.892+0000 7f40673d6640 1 -- start start 2024-03-29T11:12:41.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.892+0000 7f40673d6640 10 monclient: init 2024-03-29T11:12:41.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.893+0000 7f40673d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:41.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.893+0000 7f40673d6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:12:41.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.893+0000 7f40673d6640 10 monclient(hunting): picked mon.noname-b con 0x7f406009ac40 addr v1:172.21.15.47:6790/0 2024-03-29T11:12:41.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.893+0000 7f40673d6640 10 monclient(hunting): picked mon.noname-c con 0x7f406009c870 addr v1:172.21.15.121:6789/0 2024-03-29T11:12:41.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.893+0000 7f40673d6640 10 monclient(hunting): picked mon.noname-a con 0x7f40600b4360 addr v1:172.21.15.47:6789/0 2024-03-29T11:12:41.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.893+0000 7f40673d6640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f406009dec0 con 0x7f40600b4360 2024-03-29T11:12:41.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.893+0000 7f40673d6640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f406009da30 con 0x7f406009c870 2024-03-29T11:12:41.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.893+0000 7f40673d6640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40600b39c0 con 0x7f406009ac40 2024-03-29T11:12:41.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.893+0000 7f40673d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:41.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.893+0000 7f40673d6640 10 monclient(hunting): authenticate will time out at 816.059349s 2024-03-29T11:12:41.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.893+0000 7f4064971640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f406009c870 0x7f406013f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:54640/0 (socket says 172.21.15.121:54640) 2024-03-29T11:12:41.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.893+0000 7f4064971640 1 -- 172.21.15.121:0/1764030662 learned_addr learned my addr 172.21.15.121:0/1764030662 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-03-29T11:12:41.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.893+0000 7f4057fff640 1 -- 172.21.15.121:0/1764030662 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3906639178 0 0) 0x7f406009da30 con 0x7f406009c870 2024-03-29T11:12:41.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.893+0000 7f4057fff640 10 monclient(hunting): _negotiate 2024-03-29T11:12:41.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.893+0000 7f4057fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:41.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.893+0000 7f4057fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:41.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.893+0000 7f4057fff640 10 monclient(hunting): my global_id is 4114 2024-03-29T11:12:41.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.893+0000 7f4057fff640 1 -- 172.21.15.121:0/1764030662 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f404c003520 con 0x7f406009c870 2024-03-29T11:12:41.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.893+0000 7f4057fff640 1 -- 172.21.15.121:0/1764030662 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1194841272 0 0) 0x7f4058001490 con 0x7f40600b4360 2024-03-29T11:12:41.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.893+0000 7f4057fff640 10 monclient(hunting): _negotiate 2024-03-29T11:12:41.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.893+0000 7f4057fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:41.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.893+0000 7f4057fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:41.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.893+0000 7f4057fff640 10 monclient(hunting): my global_id is 4119 2024-03-29T11:12:41.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.893+0000 7f4057fff640 1 -- 172.21.15.121:0/1764030662 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f406009da30 con 0x7f40600b4360 2024-03-29T11:12:41.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.893+0000 7f4057fff640 1 -- 172.21.15.121:0/1764030662 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2867741949 0 0) 0x7f40600b39c0 con 0x7f406009ac40 2024-03-29T11:12:41.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.893+0000 7f4057fff640 10 monclient(hunting): _negotiate 2024-03-29T11:12:41.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.893+0000 7f4057fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:41.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.893+0000 7f4057fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:41.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.893+0000 7f4057fff640 10 monclient(hunting): my global_id is 4118 2024-03-29T11:12:41.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.893+0000 7f4057fff640 1 -- 172.21.15.121:0/1764030662 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4058001490 con 0x7f406009ac40 2024-03-29T11:12:41.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.893+0000 7f4057fff640 1 -- 172.21.15.121:0/1764030662 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4246869019 0 0) 0x7f404c003520 con 0x7f406009c870 2024-03-29T11:12:41.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.893+0000 7f4057fff640 1 -- 172.21.15.121:0/1764030662 >> v1:172.21.15.47:6790/0 conn(0x7f406009ac40 legacy=0x7f40600b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:41.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.893+0000 7f4057fff640 1 -- 172.21.15.121:0/1764030662 >> v1:172.21.15.47:6789/0 conn(0x7f40600b4360 legacy=0x7f406014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:41.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.893+0000 7f4057fff640 10 monclient: _finish_hunting 0 2024-03-29T11:12:41.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.893+0000 7f4057fff640 1 monclient: found mon.noname-c 2024-03-29T11:12:41.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.893+0000 7f4057fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:41.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.893+0000 7f4057fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:12:41.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.893+0000 7f4057fff640 1 -- 172.21.15.121:0/1764030662 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40600b5160 con 0x7f406009c870 2024-03-29T11:12:41.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.893+0000 7f4057fff640 10 monclient: _finish_auth 0 2024-03-29T11:12:41.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.893+0000 7f4057fff640 10 monclient: _check_auth_tickets 2024-03-29T11:12:41.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.893+0000 7f4057fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:41.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.893+0000 7f40673d6640 5 monclient: authenticate success, global_id 4114 2024-03-29T11:12:41.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.893+0000 7f4057fff640 1 -- 172.21.15.121:0/1764030662 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f405c001690 con 0x7f406009c870 2024-03-29T11:12:41.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.893+0000 7f4057fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:41.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.893+0000 7f4057fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:12:41.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.893+0000 7f4057fff640 10 monclient: dump: 2024-03-29T11:12:41.884 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:12:41.884 INFO:teuthology.orchestra.run.smithi121.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:41.884 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:41.884 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:41.884 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:41.884 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:41.884 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:41.884 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:41.884 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:41.884 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:41.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.893+0000 7f40673d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:41.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.893+0000 7f4057fff640 1 -- 172.21.15.121:0/1764030662 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f405c001870 con 0x7f406009c870 2024-03-29T11:12:41.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.893+0000 7f4057fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:41.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.893+0000 7f4057fff640 1 -- 172.21.15.121:0/1764030662 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f405c001c80 con 0x7f406009c870 2024-03-29T11:12:41.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.893+0000 7f40673d6640 10 monclient: get_monmap_and_config success 2024-03-29T11:12:41.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.893+0000 7f4057fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:41.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.893+0000 7f40673d6640 4 set_mon_vals no callback set 2024-03-29T11:12:41.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.893+0000 7f4057fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:41.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.893+0000 7f4057fff640 10 monclient: dump: 2024-03-29T11:12:41.885 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:12:41.885 INFO:teuthology.orchestra.run.smithi121.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:41.885 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:41.885 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:41.885 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:41.885 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:41.885 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:41.886 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:41.886 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:41.886 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:41.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.894+0000 7f40673d6640 10 monclient: shutdown 2024-03-29T11:12:41.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.894+0000 7f40673d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:41.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.894+0000 7f40673d6640 1 -- 172.21.15.121:0/1764030662 >> v1:172.21.15.121:6789/0 conn(0x7f406009c870 legacy=0x7f406013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:41.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.894+0000 7f40673d6640 1 -- 172.21.15.121:0/1764030662 shutdown_connections 2024-03-29T11:12:41.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.894+0000 7f40673d6640 1 -- 172.21.15.121:0/1764030662 >> 172.21.15.121:0/1764030662 conn(0x7f4060066b00 msgr2=0x7f4060066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:41.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.894+0000 7f40673d6640 1 -- 172.21.15.121:0/1764030662 shutdown_connections 2024-03-29T11:12:41.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.894+0000 7f40673d6640 1 -- 172.21.15.121:0/1764030662 wait complete. 2024-03-29T11:12:41.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.895+0000 7f40673d6640 10 monclient: build_initial_monmap 2024-03-29T11:12:41.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.895+0000 7f40673d6640 10 monclient: monmap: 2024-03-29T11:12:41.886 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-03-29T11:12:41.886 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:12:41.887 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-03-29T11:12:41.887 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-03-29T11:12:41.887 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:41.887 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:41.887 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:12:41.887 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:12:41.887 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:12:41.887 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:41.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.895+0000 7f40673d6640 1 librados: starting msgr at 2024-03-29T11:12:41.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.895+0000 7f40673d6640 1 librados: starting objecter 2024-03-29T11:12:41.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.895+0000 7f40673d6640 1 Processor -- start 2024-03-29T11:12:41.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.895+0000 7f40673d6640 1 -- start start 2024-03-29T11:12:41.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.895+0000 7f40673d6640 1 librados: setting wanted keys 2024-03-29T11:12:41.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.895+0000 7f40673d6640 1 librados: calling monclient init 2024-03-29T11:12:41.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.895+0000 7f40673d6640 10 monclient: init 2024-03-29T11:12:41.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.895+0000 7f40673d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:41.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f40673d6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:12:41.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f40673d6640 10 monclient(hunting): picked mon.noname-a con 0x7f406009ac40 addr v1:172.21.15.47:6789/0 2024-03-29T11:12:41.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f40673d6640 10 monclient(hunting): picked mon.noname-b con 0x7f40600b6b00 addr v1:172.21.15.121:6789/0 2024-03-29T11:12:41.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f40673d6640 10 monclient(hunting): picked mon.noname-c con 0x7f40601406b0 addr v1:172.21.15.47:6790/0 2024-03-29T11:12:41.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f40673d6640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40601d7e20 con 0x7f406009ac40 2024-03-29T11:12:41.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f40673d6640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4060140b00 con 0x7f40600b6b00 2024-03-29T11:12:41.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f40673d6640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40600ad080 con 0x7f40601406b0 2024-03-29T11:12:41.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f40673d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:41.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f40673d6640 10 monclient(hunting): authenticate will time out at 816.062180s 2024-03-29T11:12:41.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4064971640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f40600b6b00 0x7f406013ffa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:54654/0 (socket says 172.21.15.121:54654) 2024-03-29T11:12:41.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4064971640 1 -- 172.21.15.121:0/2244890745 learned_addr learned my addr 172.21.15.121:0/2244890745 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-03-29T11:12:41.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 10 client.?.objecter ms_handle_connect 0x7f40600b6b00 2024-03-29T11:12:41.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:41.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 10 client.?.objecter ms_handle_connect 0x7f406009ac40 2024-03-29T11:12:41.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:41.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 10 client.?.objecter ms_handle_connect 0x7f40601406b0 2024-03-29T11:12:41.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:41.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 1 -- 172.21.15.121:0/2244890745 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 265630599 0 0) 0x7f4060140b00 con 0x7f40600b6b00 2024-03-29T11:12:41.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:12:41.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:41.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:41.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 10 monclient(hunting): my global_id is 4117 2024-03-29T11:12:41.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 1 -- 172.21.15.121:0/2244890745 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4044004eb0 con 0x7f40600b6b00 2024-03-29T11:12:41.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 1 -- 172.21.15.121:0/2244890745 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3818169989 0 0) 0x7f40601d7e20 con 0x7f406009ac40 2024-03-29T11:12:41.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:12:41.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:41.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:41.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 10 monclient(hunting): my global_id is 4122 2024-03-29T11:12:41.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 1 -- 172.21.15.121:0/2244890745 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4060140b00 con 0x7f406009ac40 2024-03-29T11:12:41.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 1 -- 172.21.15.121:0/2244890745 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1295211181 0 0) 0x7f40600ad080 con 0x7f40601406b0 2024-03-29T11:12:41.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:12:41.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:41.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:41.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 10 monclient(hunting): my global_id is 4121 2024-03-29T11:12:41.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 1 -- 172.21.15.121:0/2244890745 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40601d7e20 con 0x7f40601406b0 2024-03-29T11:12:41.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 1 -- 172.21.15.121:0/2244890745 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 978054167 0 0) 0x7f4044004eb0 con 0x7f40600b6b00 2024-03-29T11:12:41.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 1 -- 172.21.15.121:0/2244890745 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f40600ad080 con 0x7f40600b6b00 2024-03-29T11:12:41.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 1 -- 172.21.15.121:0/2244890745 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f405c001bc0 con 0x7f40600b6b00 2024-03-29T11:12:41.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:41.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:12:41.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 10 monclient(hunting): dump: 2024-03-29T11:12:41.890 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:12:41.891 INFO:teuthology.orchestra.run.smithi121.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:41.891 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:41.891 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:41.891 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:41.891 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:41.891 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:41.891 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:41.891 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:41.891 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:41.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:12:41.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 1 -- 172.21.15.121:0/2244890745 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1103363612 0 0) 0x7f40601d7e20 con 0x7f40601406b0 2024-03-29T11:12:41.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 1 -- 172.21.15.121:0/2244890745 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4044004eb0 con 0x7f40601406b0 2024-03-29T11:12:41.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 1 -- 172.21.15.121:0/2244890745 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f4058001a00 con 0x7f40601406b0 2024-03-29T11:12:41.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:41.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:12:41.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 10 monclient(hunting): dump: 2024-03-29T11:12:41.892 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:12:41.892 INFO:teuthology.orchestra.run.smithi121.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:41.892 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:41.892 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:41.892 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:41.892 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:41.892 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:41.892 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:41.892 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:41.892 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:41.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 1 -- 172.21.15.121:0/2244890745 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4226515696 0 0) 0x7f4060140b00 con 0x7f406009ac40 2024-03-29T11:12:41.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 1 -- 172.21.15.121:0/2244890745 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f40601d7e20 con 0x7f406009ac40 2024-03-29T11:12:41.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 1 -- 172.21.15.121:0/2244890745 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f40500017a0 con 0x7f406009ac40 2024-03-29T11:12:41.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:41.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:12:41.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 10 monclient(hunting): dump: 2024-03-29T11:12:41.893 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:12:41.893 INFO:teuthology.orchestra.run.smithi121.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:41.893 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:41.893 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:41.893 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:41.893 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:41.893 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:41.894 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:41.894 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:41.894 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:41.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 1 -- 172.21.15.121:0/2244890745 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1496233685 0 0) 0x7f40600ad080 con 0x7f40600b6b00 2024-03-29T11:12:41.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 1 -- 172.21.15.121:0/2244890745 >> v1:172.21.15.47:6790/0 conn(0x7f40601406b0 legacy=0x7f40601e1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:41.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 1 -- 172.21.15.121:0/2244890745 >> v1:172.21.15.47:6789/0 conn(0x7f406009ac40 legacy=0x7f40601d73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:41.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:12:41.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 1 monclient: found mon.b 2024-03-29T11:12:41.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:41.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:12:41.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 1 -- 172.21.15.121:0/2244890745 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40601d7ab0 con 0x7f40600b6b00 2024-03-29T11:12:41.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 10 monclient: _finish_auth 0 2024-03-29T11:12:41.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:12:41.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:41.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f40673d6640 5 monclient: authenticate success, global_id 4117 2024-03-29T11:12:41.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f40673d6640 10 monclient: _renew_subs 2024-03-29T11:12:41.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f40673d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:12:41.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f40673d6640 1 -- 172.21.15.121:0/2244890745 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40601e3570 con 0x7f40600b6b00 2024-03-29T11:12:41.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 1 -- 172.21.15.121:0/2244890745 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f405c004bf0 con 0x7f40600b6b00 2024-03-29T11:12:41.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:41.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 1 -- 172.21.15.121:0/2244890745 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f405c004f00 con 0x7f40600b6b00 2024-03-29T11:12:41.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:41.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:41.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f4055ffb640 10 monclient: dump: 2024-03-29T11:12:41.896 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:12:41.896 INFO:teuthology.orchestra.run.smithi121.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:41.896 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:41.896 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:41.896 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:41.896 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:41.896 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:41.896 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:41.896 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:41.896 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:41.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.896+0000 7f40577fe640 4 set_mon_vals no callback set 2024-03-29T11:12:41.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.897+0000 7f40673d6640 10 client.4117.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:12:41.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.897+0000 7f40673d6640 10 monclient: _renew_subs 2024-03-29T11:12:41.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.897+0000 7f40673d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:12:41.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.897+0000 7f4055ffb640 1 -- 172.21.15.121:0/2244890745 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f405c0053f0 con 0x7f40600b6b00 2024-03-29T11:12:41.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.897+0000 7f40673d6640 1 -- 172.21.15.121:0/2244890745 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f40601e38e0 con 0x7f40600b6b00 2024-03-29T11:12:41.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.897+0000 7f40673d6640 1 librados: init done 2024-03-29T11:12:41.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.897+0000 7f4055ffb640 1 -- 172.21.15.121:0/2244890745 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 4053400769 0 0) 0x7f405c005a50 con 0x7f40600b6b00 2024-03-29T11:12:41.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.897+0000 7f4055ffb640 10 client.4117.objecter ms_dispatch 0x7f4060000b90 osd_map(3..3 src has 1..3) v4 2024-03-29T11:12:41.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.897+0000 7f4055ffb640 3 client.4117.objecter handle_osd_map got epochs [3,3] > 0 2024-03-29T11:12:41.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.897+0000 7f4055ffb640 3 client.4117.objecter handle_osd_map decoding full epoch 3 2024-03-29T11:12:41.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.897+0000 7f4055ffb640 20 client.4117.objecter dump_active .. 0 homeless 2024-03-29T11:12:41.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.897+0000 7f40673d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:41.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.897+0000 7f40673d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:41.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.897+0000 7f40673d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:12:41.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.897+0000 7f40673d6640 1 -- 172.21.15.121:0/2244890745 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f406009b020 con 0x7f40600b6b00 2024-03-29T11:12:41.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.899+0000 7f4055ffb640 1 -- 172.21.15.121:0/2244890745 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f405c005c10 con 0x7f40600b6b00 2024-03-29T11:12:41.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.899+0000 7f4055ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:41.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.899+0000 7f4055ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:12:41.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.992+0000 7f40673d6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "bdab7243-0401-4284-ac2d-2802e712c07b", "id": 1}] 2024-03-29T11:12:41.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.992+0000 7f40673d6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "bdab7243-0401-4284-ac2d-2802e712c07b", "id": 1}] 2024-03-29T11:12:41.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.992+0000 7f40673d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:12:41.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.992+0000 7f40673d6640 1 -- 172.21.15.121:0/2244890745 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd new", "uuid": "bdab7243-0401-4284-ac2d-2802e712c07b", "id": 1} v 0) v1 -- 0x7f406004f730 con 0x7f40600b6b00 2024-03-29T11:12:41.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.997+0000 7f4055ffb640 1 -- 172.21.15.121:0/2244890745 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "bdab7243-0401-4284-ac2d-2802e712c07b", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 4036109524 0 940483276) 0x7f405c005db0 con 0x7f40600b6b00 2024-03-29T11:12:41.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.997+0000 7f4055ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "bdab7243-0401-4284-ac2d-2802e712c07b", "id": 1}] 2024-03-29T11:12:41.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.997+0000 7f4055ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:12:41.975 INFO:teuthology.orchestra.run.smithi121.stdout:1 2024-03-29T11:12:41.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.998+0000 7f40673d6640 10 librados: watch_flush enter 2024-03-29T11:12:41.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.998+0000 7f40673d6640 10 librados: watch_flush exit 2024-03-29T11:12:41.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.998+0000 7f40673d6640 20 client.4117.objecter shutdown clearing up homeless session... 2024-03-29T11:12:41.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.998+0000 7f40673d6640 10 client.4117.objecter successfully canceled tick 2024-03-29T11:12:41.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.998+0000 7f40673d6640 10 monclient: shutdown 2024-03-29T11:12:41.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.998+0000 7f40673d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:41.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.998+0000 7f40673d6640 1 -- 172.21.15.121:0/2244890745 >> v1:172.21.15.121:6789/0 conn(0x7f40600b6b00 legacy=0x7f406013ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:41.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.998+0000 7f40673d6640 1 -- 172.21.15.121:0/2244890745 shutdown_connections 2024-03-29T11:12:41.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.998+0000 7f40673d6640 1 -- 172.21.15.121:0/2244890745 >> 172.21.15.121:0/2244890745 conn(0x7f4060066b00 msgr2=0x7f40600b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:41.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.998+0000 7f40673d6640 1 -- 172.21.15.121:0/2244890745 shutdown_connections 2024-03-29T11:12:41.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.998+0000 7f40673d6640 1 -- 172.21.15.121:0/2244890745 wait complete. 2024-03-29T11:12:41.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:41.998+0000 7f40673d6640 1 librados: shutdown 2024-03-29T11:12:41.987 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new 5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c 2 2024-03-29T11:12:42.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.109+0000 7f7c1d95d640 10 monclient: get_monmap_and_config 2024-03-29T11:12:42.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.109+0000 7f7c1d95d640 10 monclient: build_initial_monmap 2024-03-29T11:12:42.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.109+0000 7f7c1d95d640 10 monclient: monmap: 2024-03-29T11:12:42.087 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-03-29T11:12:42.087 INFO:teuthology.orchestra.run.smithi121.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:42.087 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:12:42.110169+0000 2024-03-29T11:12:42.087 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:12:42.110169+0000 2024-03-29T11:12:42.087 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:42.087 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:42.087 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:12:42.088 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:12:42.088 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:12:42.088 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:42.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.109+0000 7f7c1d95d640 1 Processor -- start 2024-03-29T11:12:42.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.109+0000 7f7c1d95d640 1 -- start start 2024-03-29T11:12:42.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.109+0000 7f7c1d95d640 10 monclient: init 2024-03-29T11:12:42.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.110+0000 7f7c1d95d640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:42.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.110+0000 7f7c1d95d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:12:42.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.110+0000 7f7c1d95d640 10 monclient(hunting): picked mon.noname-b con 0x7f7c180b1780 addr v1:172.21.15.47:6790/0 2024-03-29T11:12:42.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.110+0000 7f7c1d95d640 10 monclient(hunting): picked mon.noname-a con 0x7f7c1813d7e0 addr v1:172.21.15.47:6789/0 2024-03-29T11:12:42.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.110+0000 7f7c1d95d640 10 monclient(hunting): picked mon.noname-c con 0x7f7c18143be0 addr v1:172.21.15.121:6789/0 2024-03-29T11:12:42.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.110+0000 7f7c1d95d640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c18070b30 con 0x7f7c1813d7e0 2024-03-29T11:12:42.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.110+0000 7f7c1d95d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c18070da0 con 0x7f7c18143be0 2024-03-29T11:12:42.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.110+0000 7f7c1d95d640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c1809adb0 con 0x7f7c180b1780 2024-03-29T11:12:42.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.110+0000 7f7c1d95d640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:42.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.110+0000 7f7c1d95d640 10 monclient(hunting): authenticate will time out at 816.276556s 2024-03-29T11:12:42.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.110+0000 7f7c177fe640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f7c18143be0 0x7f7c1814b780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:54656/0 (socket says 172.21.15.121:54656) 2024-03-29T11:12:42.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.110+0000 7f7c177fe640 1 -- 172.21.15.121:0/2969555411 learned_addr learned my addr 172.21.15.121:0/2969555411 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-03-29T11:12:42.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.110+0000 7f7c15ffb640 1 -- 172.21.15.121:0/2969555411 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1670151973 0 0) 0x7f7c18070da0 con 0x7f7c18143be0 2024-03-29T11:12:42.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.110+0000 7f7c15ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:12:42.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.110+0000 7f7c15ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:42.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.110+0000 7f7c15ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:42.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.110+0000 7f7c15ffb640 10 monclient(hunting): my global_id is 4120 2024-03-29T11:12:42.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.110+0000 7f7c15ffb640 1 -- 172.21.15.121:0/2969555411 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c00003520 con 0x7f7c18143be0 2024-03-29T11:12:42.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.110+0000 7f7c15ffb640 1 -- 172.21.15.121:0/2969555411 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 650356716 0 0) 0x7f7c1809adb0 con 0x7f7c180b1780 2024-03-29T11:12:42.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.110+0000 7f7c15ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:12:42.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.110+0000 7f7c15ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:42.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.110+0000 7f7c15ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:42.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.110+0000 7f7c15ffb640 10 monclient(hunting): my global_id is 4124 2024-03-29T11:12:42.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.110+0000 7f7c15ffb640 1 -- 172.21.15.121:0/2969555411 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c18070da0 con 0x7f7c180b1780 2024-03-29T11:12:42.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.110+0000 7f7c15ffb640 1 -- 172.21.15.121:0/2969555411 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3557675378 0 0) 0x7f7c18070b30 con 0x7f7c1813d7e0 2024-03-29T11:12:42.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.110+0000 7f7c15ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:12:42.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.110+0000 7f7c15ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:42.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.110+0000 7f7c15ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:42.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.110+0000 7f7c15ffb640 10 monclient(hunting): my global_id is 4125 2024-03-29T11:12:42.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.110+0000 7f7c15ffb640 1 -- 172.21.15.121:0/2969555411 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c1809adb0 con 0x7f7c1813d7e0 2024-03-29T11:12:42.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.110+0000 7f7c15ffb640 1 -- 172.21.15.121:0/2969555411 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1237621181 0 0) 0x7f7c00003520 con 0x7f7c18143be0 2024-03-29T11:12:42.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.111+0000 7f7c15ffb640 1 -- 172.21.15.121:0/2969555411 >> v1:172.21.15.47:6790/0 conn(0x7f7c180b1780 legacy=0x7f7c1813d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:42.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.111+0000 7f7c15ffb640 1 -- 172.21.15.121:0/2969555411 >> v1:172.21.15.47:6789/0 conn(0x7f7c1813d7e0 legacy=0x7f7c181434d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:42.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.111+0000 7f7c15ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:12:42.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.111+0000 7f7c15ffb640 1 monclient: found mon.noname-c 2024-03-29T11:12:42.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.111+0000 7f7c15ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:42.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.111+0000 7f7c15ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:12:42.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.111+0000 7f7c15ffb640 1 -- 172.21.15.121:0/2969555411 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c18144070 con 0x7f7c18143be0 2024-03-29T11:12:42.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.111+0000 7f7c15ffb640 10 monclient: _finish_auth 0 2024-03-29T11:12:42.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.111+0000 7f7c15ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:12:42.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.111+0000 7f7c15ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:42.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.111+0000 7f7c1d95d640 5 monclient: authenticate success, global_id 4120 2024-03-29T11:12:42.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.111+0000 7f7c15ffb640 1 -- 172.21.15.121:0/2969555411 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f7c10001a10 con 0x7f7c18143be0 2024-03-29T11:12:42.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.111+0000 7f7c1d95d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:42.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.111+0000 7f7c15ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:42.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.111+0000 7f7c15ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:12:42.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.111+0000 7f7c15ffb640 10 monclient: dump: 2024-03-29T11:12:42.100 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:12:42.100 INFO:teuthology.orchestra.run.smithi121.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:42.100 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:42.101 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:42.101 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:42.101 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:42.101 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:42.101 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:42.101 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:42.101 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:42.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.111+0000 7f7c1d95d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:42.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.111+0000 7f7c15ffb640 1 -- 172.21.15.121:0/2969555411 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7c10001bf0 con 0x7f7c18143be0 2024-03-29T11:12:42.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.111+0000 7f7c15ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:42.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.111+0000 7f7c1d95d640 10 monclient: get_monmap_and_config success 2024-03-29T11:12:42.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.111+0000 7f7c1d95d640 4 set_mon_vals no callback set 2024-03-29T11:12:42.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.111+0000 7f7c15ffb640 1 -- 172.21.15.121:0/2969555411 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f7c10002000 con 0x7f7c18143be0 2024-03-29T11:12:42.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.111+0000 7f7c15ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:42.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.111+0000 7f7c15ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:42.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.111+0000 7f7c15ffb640 10 monclient: dump: 2024-03-29T11:12:42.102 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:12:42.102 INFO:teuthology.orchestra.run.smithi121.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:42.102 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:42.102 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:42.102 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:42.102 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:42.102 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:42.103 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:42.103 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:42.103 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:42.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.111+0000 7f7c1d95d640 10 monclient: shutdown 2024-03-29T11:12:42.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.111+0000 7f7c1d95d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:42.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.111+0000 7f7c1d95d640 1 -- 172.21.15.121:0/2969555411 >> v1:172.21.15.121:6789/0 conn(0x7f7c18143be0 legacy=0x7f7c1814b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:42.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.111+0000 7f7c1d95d640 1 -- 172.21.15.121:0/2969555411 shutdown_connections 2024-03-29T11:12:42.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.111+0000 7f7c1d95d640 1 -- 172.21.15.121:0/2969555411 >> 172.21.15.121:0/2969555411 conn(0x7f7c180b58f0 msgr2=0x7f7c180b5d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:42.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.111+0000 7f7c1d95d640 1 -- 172.21.15.121:0/2969555411 shutdown_connections 2024-03-29T11:12:42.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.111+0000 7f7c1d95d640 1 -- 172.21.15.121:0/2969555411 wait complete. 2024-03-29T11:12:42.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.112+0000 7f7c1d95d640 10 monclient: build_initial_monmap 2024-03-29T11:12:42.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.112+0000 7f7c1d95d640 10 monclient: monmap: 2024-03-29T11:12:42.104 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-03-29T11:12:42.104 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:12:42.104 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-03-29T11:12:42.104 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-03-29T11:12:42.104 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:42.104 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:42.104 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:12:42.104 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:12:42.104 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:12:42.104 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:42.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.112+0000 7f7c1d95d640 1 librados: starting msgr at 2024-03-29T11:12:42.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.112+0000 7f7c1d95d640 1 librados: starting objecter 2024-03-29T11:12:42.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.112+0000 7f7c1d95d640 1 Processor -- start 2024-03-29T11:12:42.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.112+0000 7f7c1d95d640 1 -- start start 2024-03-29T11:12:42.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.112+0000 7f7c1d95d640 1 librados: setting wanted keys 2024-03-29T11:12:42.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.112+0000 7f7c1d95d640 1 librados: calling monclient init 2024-03-29T11:12:42.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.112+0000 7f7c1d95d640 10 monclient: init 2024-03-29T11:12:42.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.113+0000 7f7c1d95d640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:42.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.113+0000 7f7c1d95d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:12:42.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.113+0000 7f7c1d95d640 10 monclient(hunting): picked mon.noname-a con 0x7f7c180b1780 addr v1:172.21.15.47:6789/0 2024-03-29T11:12:42.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.113+0000 7f7c1d95d640 10 monclient(hunting): picked mon.noname-c con 0x7f7c1813d7e0 addr v1:172.21.15.47:6790/0 2024-03-29T11:12:42.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.113+0000 7f7c1d95d640 10 monclient(hunting): picked mon.noname-b con 0x7f7c18143be0 addr v1:172.21.15.121:6789/0 2024-03-29T11:12:42.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.113+0000 7f7c1d95d640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c1806c200 con 0x7f7c180b1780 2024-03-29T11:12:42.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.113+0000 7f7c1d95d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c18136470 con 0x7f7c18143be0 2024-03-29T11:12:42.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.113+0000 7f7c1d95d640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c1806bce0 con 0x7f7c1813d7e0 2024-03-29T11:12:42.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.113+0000 7f7c1d95d640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:42.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.113+0000 7f7c1d95d640 10 monclient(hunting): authenticate will time out at 816.279517s 2024-03-29T11:12:42.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.113+0000 7f7c177fe640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f7c18143be0 0x7f7c18135d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:54670/0 (socket says 172.21.15.121:54670) 2024-03-29T11:12:42.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.113+0000 7f7c177fe640 1 -- 172.21.15.121:0/1129973273 learned_addr learned my addr 172.21.15.121:0/1129973273 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-03-29T11:12:42.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.113+0000 7f7bfffff640 10 client.?.objecter ms_handle_connect 0x7f7c18143be0 2024-03-29T11:12:42.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.113+0000 7f7bfffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:42.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.113+0000 7f7bfffff640 10 client.?.objecter ms_handle_connect 0x7f7c1813d7e0 2024-03-29T11:12:42.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.113+0000 7f7bfffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:42.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.113+0000 7f7bfffff640 10 client.?.objecter ms_handle_connect 0x7f7c180b1780 2024-03-29T11:12:42.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.113+0000 7f7bfffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:42.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.113+0000 7f7bfffff640 1 -- 172.21.15.121:0/1129973273 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1315883291 0 0) 0x7f7c18136470 con 0x7f7c18143be0 2024-03-29T11:12:42.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.113+0000 7f7bfffff640 10 monclient(hunting): _negotiate 2024-03-29T11:12:42.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.113+0000 7f7bfffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:42.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.113+0000 7f7bfffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:42.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.113+0000 7f7bfffff640 10 monclient(hunting): my global_id is 4123 2024-03-29T11:12:42.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.113+0000 7f7bfffff640 1 -- 172.21.15.121:0/1129973273 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c04004eb0 con 0x7f7c18143be0 2024-03-29T11:12:42.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.113+0000 7f7bfffff640 1 -- 172.21.15.121:0/1129973273 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3156920516 0 0) 0x7f7c1806bce0 con 0x7f7c1813d7e0 2024-03-29T11:12:42.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.113+0000 7f7bfffff640 10 monclient(hunting): _negotiate 2024-03-29T11:12:42.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.113+0000 7f7bfffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:42.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.113+0000 7f7bfffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:42.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.113+0000 7f7bfffff640 10 monclient(hunting): my global_id is 4127 2024-03-29T11:12:42.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.113+0000 7f7bfffff640 1 -- 172.21.15.121:0/1129973273 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c18136470 con 0x7f7c1813d7e0 2024-03-29T11:12:42.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.113+0000 7f7bfffff640 1 -- 172.21.15.121:0/1129973273 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2154474733 0 0) 0x7f7c1806c200 con 0x7f7c180b1780 2024-03-29T11:12:42.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.113+0000 7f7bfffff640 10 monclient(hunting): _negotiate 2024-03-29T11:12:42.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.113+0000 7f7bfffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:42.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.113+0000 7f7bfffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:42.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.113+0000 7f7bfffff640 10 monclient(hunting): my global_id is 4128 2024-03-29T11:12:42.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.113+0000 7f7bfffff640 1 -- 172.21.15.121:0/1129973273 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c1806bce0 con 0x7f7c180b1780 2024-03-29T11:12:42.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.113+0000 7f7bfffff640 1 -- 172.21.15.121:0/1129973273 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1545070720 0 0) 0x7f7c04004eb0 con 0x7f7c18143be0 2024-03-29T11:12:42.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.113+0000 7f7bfffff640 1 -- 172.21.15.121:0/1129973273 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7c1806c200 con 0x7f7c18143be0 2024-03-29T11:12:42.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.113+0000 7f7bfffff640 1 -- 172.21.15.121:0/1129973273 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f7c10001f40 con 0x7f7c18143be0 2024-03-29T11:12:42.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.113+0000 7f7bfffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:42.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.113+0000 7f7bfffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:12:42.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.113+0000 7f7bfffff640 10 monclient(hunting): dump: 2024-03-29T11:12:42.109 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:12:42.109 INFO:teuthology.orchestra.run.smithi121.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:42.110 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:42.110 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:42.110 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:42.110 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:42.110 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:42.110 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:42.110 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:42.110 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:42.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.113+0000 7f7bfffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:12:42.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.113+0000 7f7bfffff640 1 -- 172.21.15.121:0/1129973273 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4240507511 0 0) 0x7f7c18136470 con 0x7f7c1813d7e0 2024-03-29T11:12:42.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.113+0000 7f7bfffff640 1 -- 172.21.15.121:0/1129973273 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7c04004eb0 con 0x7f7c1813d7e0 2024-03-29T11:12:42.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.113+0000 7f7bfffff640 1 -- 172.21.15.121:0/1129973273 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f7c0c001870 con 0x7f7c1813d7e0 2024-03-29T11:12:42.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.113+0000 7f7bfffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:42.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.113+0000 7f7bfffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:12:42.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.113+0000 7f7bfffff640 10 monclient(hunting): dump: 2024-03-29T11:12:42.111 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:12:42.111 INFO:teuthology.orchestra.run.smithi121.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:42.111 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:42.111 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:42.111 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:42.111 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:42.112 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:42.112 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:42.112 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:42.112 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:42.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.114+0000 7f7bfffff640 1 -- 172.21.15.121:0/1129973273 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3692444545 0 0) 0x7f7c1806bce0 con 0x7f7c180b1780 2024-03-29T11:12:42.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.114+0000 7f7bfffff640 1 -- 172.21.15.121:0/1129973273 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7c18136470 con 0x7f7c180b1780 2024-03-29T11:12:42.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.114+0000 7f7bfffff640 1 -- 172.21.15.121:0/1129973273 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f7c08001790 con 0x7f7c180b1780 2024-03-29T11:12:42.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.114+0000 7f7bfffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:42.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.114+0000 7f7bfffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:12:42.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.114+0000 7f7bfffff640 10 monclient(hunting): dump: 2024-03-29T11:12:42.113 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:12:42.113 INFO:teuthology.orchestra.run.smithi121.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:42.113 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:42.113 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:42.113 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:42.113 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:42.113 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:42.113 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:42.114 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:42.114 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:42.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.114+0000 7f7bfffff640 1 -- 172.21.15.121:0/1129973273 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2862783013 0 0) 0x7f7c1806c200 con 0x7f7c18143be0 2024-03-29T11:12:42.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.114+0000 7f7bfffff640 1 -- 172.21.15.121:0/1129973273 >> v1:172.21.15.47:6790/0 conn(0x7f7c1813d7e0 legacy=0x7f7c18135650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:42.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.114+0000 7f7bfffff640 1 -- 172.21.15.121:0/1129973273 >> v1:172.21.15.47:6789/0 conn(0x7f7c180b1780 legacy=0x7f7c181da3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:42.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.114+0000 7f7bfffff640 10 monclient: _finish_hunting 0 2024-03-29T11:12:42.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.114+0000 7f7bfffff640 1 monclient: found mon.b 2024-03-29T11:12:42.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.114+0000 7f7bfffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:42.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.114+0000 7f7bfffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:12:42.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.114+0000 7f7bfffff640 1 -- 172.21.15.121:0/1129973273 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c1806afe0 con 0x7f7c18143be0 2024-03-29T11:12:42.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.114+0000 7f7bfffff640 10 monclient: _finish_auth 0 2024-03-29T11:12:42.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.114+0000 7f7bfffff640 10 monclient: _check_auth_tickets 2024-03-29T11:12:42.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.114+0000 7f7bfffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:42.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.114+0000 7f7c1d95d640 5 monclient: authenticate success, global_id 4123 2024-03-29T11:12:42.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.114+0000 7f7c1d95d640 10 monclient: _renew_subs 2024-03-29T11:12:42.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.114+0000 7f7c1d95d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:12:42.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.114+0000 7f7c1d95d640 1 -- 172.21.15.121:0/1129973273 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c1806c410 con 0x7f7c18143be0 2024-03-29T11:12:42.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.114+0000 7f7c1d95d640 10 client.4123.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:12:42.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.114+0000 7f7c1d95d640 10 monclient: _renew_subs 2024-03-29T11:12:42.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.114+0000 7f7c1d95d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:12:42.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.114+0000 7f7c1d95d640 1 -- 172.21.15.121:0/1129973273 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c1806ba30 con 0x7f7c18143be0 2024-03-29T11:12:42.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.114+0000 7f7c1d95d640 1 librados: init done 2024-03-29T11:12:42.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.114+0000 7f7bfffff640 1 -- 172.21.15.121:0/1129973273 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7c10004e10 con 0x7f7c18143be0 2024-03-29T11:12:42.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.114+0000 7f7bfffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:42.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.114+0000 7f7bfffff640 1 -- 172.21.15.121:0/1129973273 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f7c10005120 con 0x7f7c18143be0 2024-03-29T11:12:42.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.114+0000 7f7bfffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:42.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.114+0000 7f7bfffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:42.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.114+0000 7f7bfffff640 10 monclient: dump: 2024-03-29T11:12:42.116 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:12:42.116 INFO:teuthology.orchestra.run.smithi121.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:42.116 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:42.116 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:42.116 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:42.116 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:42.116 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:42.116 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:42.116 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:42.116 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:42.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.114+0000 7f7c157fa640 4 set_mon_vals no callback set 2024-03-29T11:12:42.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.114+0000 7f7bfffff640 1 -- 172.21.15.121:0/1129973273 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f7c10005690 con 0x7f7c18143be0 2024-03-29T11:12:42.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.114+0000 7f7bfffff640 1 -- 172.21.15.121:0/1129973273 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 3148853904 0 0) 0x7f7c10005d60 con 0x7f7c18143be0 2024-03-29T11:12:42.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.114+0000 7f7bfffff640 10 client.4123.objecter ms_dispatch 0x7f7c18000b90 osd_map(4..4 src has 1..4) v4 2024-03-29T11:12:42.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.114+0000 7f7bfffff640 3 client.4123.objecter handle_osd_map got epochs [4,4] > 0 2024-03-29T11:12:42.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.114+0000 7f7bfffff640 3 client.4123.objecter handle_osd_map decoding full epoch 4 2024-03-29T11:12:42.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.114+0000 7f7bfffff640 20 client.4123.objecter dump_active .. 0 homeless 2024-03-29T11:12:42.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.115+0000 7f7c1d95d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:42.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.115+0000 7f7c1d95d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:42.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.115+0000 7f7c1d95d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:12:42.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.115+0000 7f7c1d95d640 1 -- 172.21.15.121:0/1129973273 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bf8005080 con 0x7f7c18143be0 2024-03-29T11:12:42.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.117+0000 7f7bfffff640 1 -- 172.21.15.121:0/1129973273 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f7c10006000 con 0x7f7c18143be0 2024-03-29T11:12:42.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.117+0000 7f7bfffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:42.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.117+0000 7f7bfffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:12:42.118 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-29T11:12:42.121+0000 7fcab480f040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:12:42.134 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-29T11:12:42.157+0000 7f3465a3c040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:12:42.155 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-29T11:12:42.177+0000 7fcab480f040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:12:42.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.189+0000 7f7c1d95d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c", "id": 2}] 2024-03-29T11:12:42.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.189+0000 7f7c1d95d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c", "id": 2}] 2024-03-29T11:12:42.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.189+0000 7f7c1d95d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:12:42.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.189+0000 7f7c1d95d640 1 -- 172.21.15.121:0/1129973273 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd new", "uuid": "5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c", "id": 2} v 0) v1 -- 0x7f7bf80058b0 con 0x7f7c18143be0 2024-03-29T11:12:42.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.193+0000 7f7bfffff640 1 -- 172.21.15.121:0/1129973273 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 1937876443 0 216599125) 0x7f7c10005940 con 0x7f7c18143be0 2024-03-29T11:12:42.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.193+0000 7f7bfffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c", "id": 2}] 2024-03-29T11:12:42.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.193+0000 7f7bfffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:12:42.171 INFO:teuthology.orchestra.run.smithi121.stdout:2 2024-03-29T11:12:42.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.194+0000 7f7c1d95d640 10 librados: watch_flush enter 2024-03-29T11:12:42.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.194+0000 7f7c1d95d640 10 librados: watch_flush exit 2024-03-29T11:12:42.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.194+0000 7f7c1d95d640 20 client.4123.objecter shutdown clearing up homeless session... 2024-03-29T11:12:42.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.194+0000 7f7c1d95d640 10 client.4123.objecter successfully canceled tick 2024-03-29T11:12:42.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.194+0000 7f7c1d95d640 10 monclient: shutdown 2024-03-29T11:12:42.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.194+0000 7f7c1d95d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:42.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.194+0000 7f7c1d95d640 1 -- 172.21.15.121:0/1129973273 >> v1:172.21.15.121:6789/0 conn(0x7f7c18143be0 legacy=0x7f7c18135d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:42.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.194+0000 7f7c1d95d640 1 -- 172.21.15.121:0/1129973273 shutdown_connections 2024-03-29T11:12:42.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.194+0000 7f7c1d95d640 1 -- 172.21.15.121:0/1129973273 >> 172.21.15.121:0/1129973273 conn(0x7f7c180b58f0 msgr2=0x7f7c180b06b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:42.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.194+0000 7f7c1d95d640 1 -- 172.21.15.121:0/1129973273 shutdown_connections 2024-03-29T11:12:42.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.194+0000 7f7c1d95d640 1 -- 172.21.15.121:0/1129973273 wait complete. 2024-03-29T11:12:42.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.194+0000 7f7c1d95d640 1 librados: shutdown 2024-03-29T11:12:42.182 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new 1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8 3 2024-03-29T11:12:42.192 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-29T11:12:42.214+0000 7f3465a3c040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:12:42.202 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-29T11:12:42.224+0000 7fcab480f040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:12:42.239 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-29T11:12:42.261+0000 7f3465a3c040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:12:42.257 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-29T11:12:42.279+0000 7fcab480f040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:12:42.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.304+0000 7fd3ff5c4640 10 monclient: get_monmap_and_config 2024-03-29T11:12:42.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.304+0000 7fd3ff5c4640 10 monclient: build_initial_monmap 2024-03-29T11:12:42.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.304+0000 7fd3ff5c4640 10 monclient: monmap: 2024-03-29T11:12:42.282 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-03-29T11:12:42.282 INFO:teuthology.orchestra.run.smithi121.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:42.282 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:12:42.304989+0000 2024-03-29T11:12:42.282 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:12:42.304989+0000 2024-03-29T11:12:42.282 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:42.282 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:42.283 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:12:42.283 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:12:42.283 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:12:42.283 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:42.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.304+0000 7fd3ff5c4640 1 Processor -- start 2024-03-29T11:12:42.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.304+0000 7fd3ff5c4640 1 -- start start 2024-03-29T11:12:42.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.304+0000 7fd3ff5c4640 10 monclient: init 2024-03-29T11:12:42.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.305+0000 7fd3ff5c4640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:42.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.305+0000 7fd3ff5c4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:12:42.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.305+0000 7fd3ff5c4640 10 monclient(hunting): picked mon.noname-c con 0x7fd3f809ac40 addr v1:172.21.15.121:6789/0 2024-03-29T11:12:42.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.305+0000 7fd3ff5c4640 10 monclient(hunting): picked mon.noname-a con 0x7fd3f809c870 addr v1:172.21.15.47:6789/0 2024-03-29T11:12:42.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.305+0000 7fd3ff5c4640 10 monclient(hunting): picked mon.noname-b con 0x7fd3f80b4360 addr v1:172.21.15.47:6790/0 2024-03-29T11:12:42.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.305+0000 7fd3ff5c4640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3f809dec0 con 0x7fd3f809c870 2024-03-29T11:12:42.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.305+0000 7fd3ff5c4640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3f809da30 con 0x7fd3f809ac40 2024-03-29T11:12:42.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.305+0000 7fd3ff5c4640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3f80b39c0 con 0x7fd3f80b4360 2024-03-29T11:12:42.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.305+0000 7fd3ff5c4640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:42.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.305+0000 7fd3ff5c4640 10 monclient(hunting): authenticate will time out at 816.471543s 2024-03-29T11:12:42.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.305+0000 7fd3fd360640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fd3f809ac40 0x7fd3f80b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:54674/0 (socket says 172.21.15.121:54674) 2024-03-29T11:12:42.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.305+0000 7fd3fd360640 1 -- 172.21.15.121:0/4083586477 learned_addr learned my addr 172.21.15.121:0/4083586477 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-03-29T11:12:42.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.305+0000 7fd3effff640 1 -- 172.21.15.121:0/4083586477 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4277276056 0 0) 0x7fd3f809da30 con 0x7fd3f809ac40 2024-03-29T11:12:42.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.305+0000 7fd3effff640 10 monclient(hunting): _negotiate 2024-03-29T11:12:42.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.305+0000 7fd3effff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:42.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.305+0000 7fd3effff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:42.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.305+0000 7fd3effff640 10 monclient(hunting): my global_id is 4126 2024-03-29T11:12:42.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.305+0000 7fd3effff640 1 -- 172.21.15.121:0/4083586477 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3e4003520 con 0x7fd3f809ac40 2024-03-29T11:12:42.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.305+0000 7fd3effff640 1 -- 172.21.15.121:0/4083586477 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3567235546 0 0) 0x7fd3f80b39c0 con 0x7fd3f80b4360 2024-03-29T11:12:42.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.305+0000 7fd3effff640 10 monclient(hunting): _negotiate 2024-03-29T11:12:42.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.305+0000 7fd3effff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:42.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.305+0000 7fd3effff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:42.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.305+0000 7fd3effff640 10 monclient(hunting): my global_id is 4130 2024-03-29T11:12:42.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.305+0000 7fd3effff640 1 -- 172.21.15.121:0/4083586477 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3f809da30 con 0x7fd3f80b4360 2024-03-29T11:12:42.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.305+0000 7fd3effff640 1 -- 172.21.15.121:0/4083586477 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3469810127 0 0) 0x7fd3f4001110 con 0x7fd3f809c870 2024-03-29T11:12:42.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.305+0000 7fd3effff640 10 monclient(hunting): _negotiate 2024-03-29T11:12:42.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.305+0000 7fd3effff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:42.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.305+0000 7fd3effff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:42.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.305+0000 7fd3effff640 10 monclient(hunting): my global_id is 4131 2024-03-29T11:12:42.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.305+0000 7fd3effff640 1 -- 172.21.15.121:0/4083586477 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3f80b39c0 con 0x7fd3f809c870 2024-03-29T11:12:42.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.306+0000 7fd3effff640 1 -- 172.21.15.121:0/4083586477 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2644681675 0 0) 0x7fd3e4003520 con 0x7fd3f809ac40 2024-03-29T11:12:42.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.306+0000 7fd3effff640 1 -- 172.21.15.121:0/4083586477 >> v1:172.21.15.47:6790/0 conn(0x7fd3f80b4360 legacy=0x7fd3f814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:42.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.306+0000 7fd3effff640 1 -- 172.21.15.121:0/4083586477 >> v1:172.21.15.47:6789/0 conn(0x7fd3f809c870 legacy=0x7fd3f813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:42.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.306+0000 7fd3effff640 10 monclient: _finish_hunting 0 2024-03-29T11:12:42.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.306+0000 7fd3effff640 1 monclient: found mon.noname-c 2024-03-29T11:12:42.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.306+0000 7fd3effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:42.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.306+0000 7fd3effff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:12:42.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.306+0000 7fd3effff640 1 -- 172.21.15.121:0/4083586477 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3f80b5160 con 0x7fd3f809ac40 2024-03-29T11:12:42.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.306+0000 7fd3effff640 10 monclient: _finish_auth 0 2024-03-29T11:12:42.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.306+0000 7fd3effff640 10 monclient: _check_auth_tickets 2024-03-29T11:12:42.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.306+0000 7fd3effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:42.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.306+0000 7fd3ff5c4640 5 monclient: authenticate success, global_id 4126 2024-03-29T11:12:42.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.306+0000 7fd3effff640 1 -- 172.21.15.121:0/4083586477 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fd3e8001690 con 0x7fd3f809ac40 2024-03-29T11:12:42.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.306+0000 7fd3ff5c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:42.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.306+0000 7fd3effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:42.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.306+0000 7fd3effff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:12:42.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.306+0000 7fd3effff640 10 monclient: dump: 2024-03-29T11:12:42.299 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:12:42.299 INFO:teuthology.orchestra.run.smithi121.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:42.299 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:42.299 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:42.299 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:42.299 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:42.299 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:42.300 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:42.300 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:42.300 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:42.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.306+0000 7fd3ff5c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:42.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.306+0000 7fd3effff640 1 -- 172.21.15.121:0/4083586477 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd3e8001870 con 0x7fd3f809ac40 2024-03-29T11:12:42.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.306+0000 7fd3effff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:42.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.306+0000 7fd3effff640 1 -- 172.21.15.121:0/4083586477 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fd3e8001c80 con 0x7fd3f809ac40 2024-03-29T11:12:42.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.306+0000 7fd3effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:42.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.306+0000 7fd3effff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:42.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.306+0000 7fd3effff640 10 monclient: dump: 2024-03-29T11:12:42.300 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:12:42.300 INFO:teuthology.orchestra.run.smithi121.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:42.300 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:42.300 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:42.301 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:42.301 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:42.301 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:42.301 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:42.301 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:42.301 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:42.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.306+0000 7fd3ff5c4640 10 monclient: get_monmap_and_config success 2024-03-29T11:12:42.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.306+0000 7fd3ff5c4640 4 set_mon_vals no callback set 2024-03-29T11:12:42.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.306+0000 7fd3ff5c4640 10 monclient: shutdown 2024-03-29T11:12:42.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.306+0000 7fd3ff5c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:42.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.306+0000 7fd3ff5c4640 1 -- 172.21.15.121:0/4083586477 >> v1:172.21.15.121:6789/0 conn(0x7fd3f809ac40 legacy=0x7fd3f80b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:42.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.306+0000 7fd3ff5c4640 1 -- 172.21.15.121:0/4083586477 shutdown_connections 2024-03-29T11:12:42.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.306+0000 7fd3ff5c4640 1 -- 172.21.15.121:0/4083586477 >> 172.21.15.121:0/4083586477 conn(0x7fd3f8066b00 msgr2=0x7fd3f8066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:42.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.306+0000 7fd3ff5c4640 1 -- 172.21.15.121:0/4083586477 shutdown_connections 2024-03-29T11:12:42.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.306+0000 7fd3ff5c4640 1 -- 172.21.15.121:0/4083586477 wait complete. 2024-03-29T11:12:42.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.307+0000 7fd3ff5c4640 10 monclient: build_initial_monmap 2024-03-29T11:12:42.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.307+0000 7fd3ff5c4640 10 monclient: monmap: 2024-03-29T11:12:42.302 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-03-29T11:12:42.302 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:12:42.302 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-03-29T11:12:42.302 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-03-29T11:12:42.303 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:42.303 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:42.303 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:12:42.303 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:12:42.303 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:12:42.303 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:42.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.307+0000 7fd3ff5c4640 1 librados: starting msgr at 2024-03-29T11:12:42.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.307+0000 7fd3ff5c4640 1 librados: starting objecter 2024-03-29T11:12:42.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.307+0000 7fd3ff5c4640 1 Processor -- start 2024-03-29T11:12:42.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.307+0000 7fd3ff5c4640 1 -- start start 2024-03-29T11:12:42.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.307+0000 7fd3ff5c4640 1 librados: setting wanted keys 2024-03-29T11:12:42.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.307+0000 7fd3ff5c4640 1 librados: calling monclient init 2024-03-29T11:12:42.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.307+0000 7fd3ff5c4640 10 monclient: init 2024-03-29T11:12:42.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.308+0000 7fd3ff5c4640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:42.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.308+0000 7fd3ff5c4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:12:42.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.308+0000 7fd3ff5c4640 10 monclient(hunting): picked mon.noname-c con 0x7fd3f809ac40 addr v1:172.21.15.47:6790/0 2024-03-29T11:12:42.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.308+0000 7fd3ff5c4640 10 monclient(hunting): picked mon.noname-a con 0x7fd3f8138f60 addr v1:172.21.15.47:6789/0 2024-03-29T11:12:42.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.308+0000 7fd3ff5c4640 10 monclient(hunting): picked mon.noname-b con 0x7fd3f80afdc0 addr v1:172.21.15.121:6789/0 2024-03-29T11:12:42.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.308+0000 7fd3ff5c4640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3f81db610 con 0x7fd3f8138f60 2024-03-29T11:12:42.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.308+0000 7fd3ff5c4640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3f8139a00 con 0x7fd3f80afdc0 2024-03-29T11:12:42.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.308+0000 7fd3ff5c4640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3f80b0230 con 0x7fd3f809ac40 2024-03-29T11:12:42.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.308+0000 7fd3ff5c4640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:42.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.308+0000 7fd3ff5c4640 10 monclient(hunting): authenticate will time out at 816.474545s 2024-03-29T11:12:42.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.308+0000 7fd3fdb61640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fd3f80afdc0 0x7fd3f81e1fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:54680/0 (socket says 172.21.15.121:54680) 2024-03-29T11:12:42.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.308+0000 7fd3fdb61640 1 -- 172.21.15.121:0/2857840772 learned_addr learned my addr 172.21.15.121:0/2857840772 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-03-29T11:12:42.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.308+0000 7fd3edffb640 10 client.?.objecter ms_handle_connect 0x7fd3f80afdc0 2024-03-29T11:12:42.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.308+0000 7fd3edffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:42.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.308+0000 7fd3edffb640 10 client.?.objecter ms_handle_connect 0x7fd3f8138f60 2024-03-29T11:12:42.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.308+0000 7fd3edffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:42.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.308+0000 7fd3edffb640 10 client.?.objecter ms_handle_connect 0x7fd3f809ac40 2024-03-29T11:12:42.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.308+0000 7fd3edffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:42.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.308+0000 7fd3edffb640 1 -- 172.21.15.121:0/2857840772 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3084829103 0 0) 0x7fd3f8139a00 con 0x7fd3f80afdc0 2024-03-29T11:12:42.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.308+0000 7fd3edffb640 10 monclient(hunting): _negotiate 2024-03-29T11:12:42.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.308+0000 7fd3edffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:42.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.308+0000 7fd3edffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:42.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.308+0000 7fd3edffb640 10 monclient(hunting): my global_id is 4129 2024-03-29T11:12:42.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.308+0000 7fd3edffb640 1 -- 172.21.15.121:0/2857840772 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3dc004eb0 con 0x7fd3f80afdc0 2024-03-29T11:12:42.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.308+0000 7fd3edffb640 1 -- 172.21.15.121:0/2857840772 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4120998487 0 0) 0x7fd3f81db610 con 0x7fd3f8138f60 2024-03-29T11:12:42.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.308+0000 7fd3edffb640 10 monclient(hunting): _negotiate 2024-03-29T11:12:42.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.308+0000 7fd3edffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:42.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.308+0000 7fd3edffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:42.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.308+0000 7fd3edffb640 10 monclient(hunting): my global_id is 4134 2024-03-29T11:12:42.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.308+0000 7fd3edffb640 1 -- 172.21.15.121:0/2857840772 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3f8139a00 con 0x7fd3f8138f60 2024-03-29T11:12:42.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.308+0000 7fd3edffb640 1 -- 172.21.15.121:0/2857840772 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3716739609 0 0) 0x7fd3f80b0230 con 0x7fd3f809ac40 2024-03-29T11:12:42.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.308+0000 7fd3edffb640 10 monclient(hunting): _negotiate 2024-03-29T11:12:42.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.308+0000 7fd3edffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:42.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.308+0000 7fd3edffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:42.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.308+0000 7fd3edffb640 10 monclient(hunting): my global_id is 4133 2024-03-29T11:12:42.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.308+0000 7fd3edffb640 1 -- 172.21.15.121:0/2857840772 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3f81db610 con 0x7fd3f809ac40 2024-03-29T11:12:42.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.308+0000 7fd3edffb640 1 -- 172.21.15.121:0/2857840772 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3970468129 0 0) 0x7fd3dc004eb0 con 0x7fd3f80afdc0 2024-03-29T11:12:42.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.309+0000 7fd3edffb640 1 -- 172.21.15.121:0/2857840772 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd3f80b0230 con 0x7fd3f80afdc0 2024-03-29T11:12:42.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.309+0000 7fd3edffb640 1 -- 172.21.15.121:0/2857840772 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fd3f0001c00 con 0x7fd3f80afdc0 2024-03-29T11:12:42.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.309+0000 7fd3edffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:42.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.309+0000 7fd3edffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:12:42.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.309+0000 7fd3edffb640 10 monclient(hunting): dump: 2024-03-29T11:12:42.307 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:12:42.308 INFO:teuthology.orchestra.run.smithi121.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:42.308 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:42.308 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:42.308 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:42.308 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:42.308 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:42.308 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:42.308 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:42.308 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:42.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.309+0000 7fd3edffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:12:42.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.309+0000 7fd3edffb640 1 -- 172.21.15.121:0/2857840772 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1923708173 0 0) 0x7fd3f8139a00 con 0x7fd3f8138f60 2024-03-29T11:12:42.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.309+0000 7fd3edffb640 1 -- 172.21.15.121:0/2857840772 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd3dc004eb0 con 0x7fd3f8138f60 2024-03-29T11:12:42.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.309+0000 7fd3edffb640 1 -- 172.21.15.121:0/2857840772 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fd3f4001860 con 0x7fd3f8138f60 2024-03-29T11:12:42.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.309+0000 7fd3edffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:42.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.309+0000 7fd3edffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:12:42.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.309+0000 7fd3edffb640 10 monclient(hunting): dump: 2024-03-29T11:12:42.309 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:12:42.309 INFO:teuthology.orchestra.run.smithi121.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:42.309 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:42.309 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:42.309 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:42.309 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:42.310 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:42.310 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:42.310 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:42.310 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:42.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.309+0000 7fd3edffb640 1 -- 172.21.15.121:0/2857840772 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1106078877 0 0) 0x7fd3f81db610 con 0x7fd3f809ac40 2024-03-29T11:12:42.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.309+0000 7fd3edffb640 1 -- 172.21.15.121:0/2857840772 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd3f8139a00 con 0x7fd3f809ac40 2024-03-29T11:12:42.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.309+0000 7fd3edffb640 1 -- 172.21.15.121:0/2857840772 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fd3e8001b80 con 0x7fd3f809ac40 2024-03-29T11:12:42.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.309+0000 7fd3edffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:42.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.309+0000 7fd3edffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:12:42.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.309+0000 7fd3edffb640 10 monclient(hunting): dump: 2024-03-29T11:12:42.310 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:12:42.310 INFO:teuthology.orchestra.run.smithi121.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:42.310 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:42.310 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:42.311 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:42.311 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:42.311 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:42.311 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:42.311 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:42.311 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:42.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.309+0000 7fd3edffb640 1 -- 172.21.15.121:0/2857840772 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1924322610 0 0) 0x7fd3f80b0230 con 0x7fd3f80afdc0 2024-03-29T11:12:42.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.309+0000 7fd3edffb640 1 -- 172.21.15.121:0/2857840772 >> v1:172.21.15.47:6790/0 conn(0x7fd3f809ac40 legacy=0x7fd3f81dac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:42.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.309+0000 7fd3edffb640 1 -- 172.21.15.121:0/2857840772 >> v1:172.21.15.47:6789/0 conn(0x7fd3f8138f60 legacy=0x7fd3f80af6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:42.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.309+0000 7fd3edffb640 10 monclient: _finish_hunting 0 2024-03-29T11:12:42.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.309+0000 7fd3edffb640 1 monclient: found mon.b 2024-03-29T11:12:42.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.309+0000 7fd3edffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:42.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.309+0000 7fd3edffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:12:42.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.309+0000 7fd3edffb640 1 -- 172.21.15.121:0/2857840772 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3f80b03c0 con 0x7fd3f80afdc0 2024-03-29T11:12:42.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.309+0000 7fd3edffb640 10 monclient: _finish_auth 0 2024-03-29T11:12:42.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.309+0000 7fd3edffb640 10 monclient: _check_auth_tickets 2024-03-29T11:12:42.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.309+0000 7fd3edffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:42.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.309+0000 7fd3ff5c4640 5 monclient: authenticate success, global_id 4129 2024-03-29T11:12:42.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.309+0000 7fd3ff5c4640 10 monclient: _renew_subs 2024-03-29T11:12:42.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.309+0000 7fd3ff5c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:12:42.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.309+0000 7fd3ff5c4640 1 -- 172.21.15.121:0/2857840772 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3f81e36e0 con 0x7fd3f80afdc0 2024-03-29T11:12:42.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.309+0000 7fd3edffb640 1 -- 172.21.15.121:0/2857840772 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd3f0003990 con 0x7fd3f80afdc0 2024-03-29T11:12:42.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.309+0000 7fd3edffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:42.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.309+0000 7fd3edffb640 1 -- 172.21.15.121:0/2857840772 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fd3f0003c00 con 0x7fd3f80afdc0 2024-03-29T11:12:42.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.309+0000 7fd3edffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:42.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.309+0000 7fd3edffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:42.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.309+0000 7fd3edffb640 10 monclient: dump: 2024-03-29T11:12:42.313 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:12:42.313 INFO:teuthology.orchestra.run.smithi121.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:42.313 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:42.313 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:42.313 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:42.313 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:42.313 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:42.313 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:42.313 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:42.314 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:42.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.309+0000 7fd3edffb640 1 -- 172.21.15.121:0/2857840772 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fd3f00041b0 con 0x7fd3f80afdc0 2024-03-29T11:12:42.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.309+0000 7fd3ef7fe640 4 set_mon_vals no callback set 2024-03-29T11:12:42.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.309+0000 7fd3ff5c4640 10 client.4129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:12:42.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.309+0000 7fd3ff5c4640 10 monclient: _renew_subs 2024-03-29T11:12:42.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.309+0000 7fd3ff5c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:12:42.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.309+0000 7fd3ff5c4640 1 -- 172.21.15.121:0/2857840772 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3f81e3a50 con 0x7fd3f80afdc0 2024-03-29T11:12:42.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.309+0000 7fd3ff5c4640 1 librados: init done 2024-03-29T11:12:42.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.309+0000 7fd3edffb640 1 -- 172.21.15.121:0/2857840772 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 453561491 0 0) 0x7fd3f00046e0 con 0x7fd3f80afdc0 2024-03-29T11:12:42.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.309+0000 7fd3edffb640 10 client.4129.objecter ms_dispatch 0x7fd3f8000b90 osd_map(5..5 src has 1..5) v4 2024-03-29T11:12:42.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.309+0000 7fd3edffb640 3 client.4129.objecter handle_osd_map got epochs [5,5] > 0 2024-03-29T11:12:42.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.309+0000 7fd3edffb640 3 client.4129.objecter handle_osd_map decoding full epoch 5 2024-03-29T11:12:42.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.309+0000 7fd3edffb640 20 client.4129.objecter dump_active .. 0 homeless 2024-03-29T11:12:42.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.310+0000 7fd3ff5c4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:42.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.310+0000 7fd3ff5c4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:42.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.310+0000 7fd3ff5c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:12:42.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.310+0000 7fd3ff5c4640 1 -- 172.21.15.121:0/2857840772 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3f81db3f0 con 0x7fd3f80afdc0 2024-03-29T11:12:42.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.312+0000 7fd3edffb640 1 -- 172.21.15.121:0/2857840772 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fd3f0005bf0 con 0x7fd3f80afdc0 2024-03-29T11:12:42.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.312+0000 7fd3edffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:42.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.312+0000 7fd3edffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:12:42.316 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-29T11:12:42.326+0000 7f3465a3c040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:12:42.316 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-29T11:12:42.324+0000 7fcab480f040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:12:42.347 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-29T11:12:42.370+0000 7f3465a3c040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:12:42.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.384+0000 7fd3ff5c4640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8", "id": 3}] 2024-03-29T11:12:42.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.384+0000 7fd3ff5c4640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8", "id": 3}] 2024-03-29T11:12:42.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.384+0000 7fd3ff5c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:12:42.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.384+0000 7fd3ff5c4640 1 -- 172.21.15.121:0/2857840772 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd new", "uuid": "1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8", "id": 3} v 0) v1 -- 0x7fd3f804f730 con 0x7fd3f80afdc0 2024-03-29T11:12:42.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.388+0000 7fd3edffb640 1 -- 172.21.15.121:0/2857840772 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 3366627884 0 525046306) 0x7fd3f00063c0 con 0x7fd3f80afdc0 2024-03-29T11:12:42.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.388+0000 7fd3edffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8", "id": 3}] 2024-03-29T11:12:42.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.388+0000 7fd3edffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:12:42.366 INFO:teuthology.orchestra.run.smithi121.stdout:3 2024-03-29T11:12:42.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.389+0000 7fd3ff5c4640 10 librados: watch_flush enter 2024-03-29T11:12:42.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.389+0000 7fd3ff5c4640 10 librados: watch_flush exit 2024-03-29T11:12:42.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.389+0000 7fd3ff5c4640 20 client.4129.objecter shutdown clearing up homeless session... 2024-03-29T11:12:42.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.389+0000 7fd3ff5c4640 10 client.4129.objecter successfully canceled tick 2024-03-29T11:12:42.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.389+0000 7fd3ff5c4640 10 monclient: shutdown 2024-03-29T11:12:42.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.389+0000 7fd3ff5c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:42.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.389+0000 7fd3ff5c4640 1 -- 172.21.15.121:0/2857840772 >> v1:172.21.15.121:6789/0 conn(0x7fd3f80afdc0 legacy=0x7fd3f81e1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:42.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.389+0000 7fd3ff5c4640 1 -- 172.21.15.121:0/2857840772 shutdown_connections 2024-03-29T11:12:42.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.389+0000 7fd3ff5c4640 1 -- 172.21.15.121:0/2857840772 >> 172.21.15.121:0/2857840772 conn(0x7fd3f8066b00 msgr2=0x7fd3f80b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:42.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.389+0000 7fd3ff5c4640 1 -- 172.21.15.121:0/2857840772 shutdown_connections 2024-03-29T11:12:42.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.389+0000 7fd3ff5c4640 1 -- 172.21.15.121:0/2857840772 wait complete. 2024-03-29T11:12:42.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.389+0000 7fd3ff5c4640 1 librados: shutdown 2024-03-29T11:12:42.377 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new e1de122b-37a5-469f-bff2-bd60d934e4d2 4 2024-03-29T11:12:42.421 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-29T11:12:42.444+0000 7fcab480f040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:12:42.469 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-29T11:12:42.491+0000 7f3465a3c040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:12:42.494 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-29T11:12:42.516+0000 7fcab480f040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:12:42.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.539+0000 7f32af3ba640 10 monclient: get_monmap_and_config 2024-03-29T11:12:42.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.539+0000 7f32af3ba640 10 monclient: build_initial_monmap 2024-03-29T11:12:42.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.539+0000 7f32af3ba640 10 monclient: monmap: 2024-03-29T11:12:42.518 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-03-29T11:12:42.518 INFO:teuthology.orchestra.run.smithi121.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:42.518 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:12:42.540523+0000 2024-03-29T11:12:42.518 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:12:42.540523+0000 2024-03-29T11:12:42.518 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:42.518 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:42.518 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:12:42.519 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:12:42.519 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:12:42.519 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:42.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.540+0000 7f32af3ba640 1 Processor -- start 2024-03-29T11:12:42.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.540+0000 7f32af3ba640 1 -- start start 2024-03-29T11:12:42.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.540+0000 7f32af3ba640 10 monclient: init 2024-03-29T11:12:42.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.540+0000 7f32af3ba640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:42.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.540+0000 7f32af3ba640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:12:42.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.540+0000 7f32af3ba640 10 monclient(hunting): picked mon.noname-b con 0x7f32a80af360 addr v1:172.21.15.47:6790/0 2024-03-29T11:12:42.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.540+0000 7f32af3ba640 10 monclient(hunting): picked mon.noname-c con 0x7f32a806af60 addr v1:172.21.15.121:6789/0 2024-03-29T11:12:42.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.540+0000 7f32af3ba640 10 monclient(hunting): picked mon.noname-a con 0x7f32a8145e60 addr v1:172.21.15.47:6789/0 2024-03-29T11:12:42.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.540+0000 7f32af3ba640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32a80b0520 con 0x7f32a8145e60 2024-03-29T11:12:42.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.541+0000 7f32af3ba640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32a80aff20 con 0x7f32a806af60 2024-03-29T11:12:42.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.541+0000 7f32af3ba640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32a80b1da0 con 0x7f32a80af360 2024-03-29T11:12:42.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.541+0000 7f32af3ba640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:42.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.541+0000 7f32af3ba640 10 monclient(hunting): authenticate will time out at 816.707141s 2024-03-29T11:12:42.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.541+0000 7f32ac955640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f32a806af60 0x7f32a8145750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:54686/0 (socket says 172.21.15.121:54686) 2024-03-29T11:12:42.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.541+0000 7f32ac955640 1 -- 172.21.15.121:0/319836133 learned_addr learned my addr 172.21.15.121:0/319836133 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-03-29T11:12:42.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.541+0000 7f329ffff640 1 -- 172.21.15.121:0/319836133 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3644118497 0 0) 0x7f32a80aff20 con 0x7f32a806af60 2024-03-29T11:12:42.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.541+0000 7f329ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:12:42.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.541+0000 7f329ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:42.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.541+0000 7f329ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:42.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.541+0000 7f329ffff640 10 monclient(hunting): my global_id is 4132 2024-03-29T11:12:42.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.541+0000 7f329ffff640 1 -- 172.21.15.121:0/319836133 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3294003520 con 0x7f32a806af60 2024-03-29T11:12:42.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.541+0000 7f329ffff640 1 -- 172.21.15.121:0/319836133 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2755140451 0 0) 0x7f32a80b1da0 con 0x7f32a80af360 2024-03-29T11:12:42.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.541+0000 7f329ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:12:42.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.541+0000 7f329ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:42.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.541+0000 7f329ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:42.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.541+0000 7f329ffff640 10 monclient(hunting): my global_id is 4136 2024-03-29T11:12:42.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.541+0000 7f329ffff640 1 -- 172.21.15.121:0/319836133 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32a80aff20 con 0x7f32a80af360 2024-03-29T11:12:42.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.541+0000 7f329ffff640 1 -- 172.21.15.121:0/319836133 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4141958286 0 0) 0x7f32a80b0520 con 0x7f32a8145e60 2024-03-29T11:12:42.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.541+0000 7f329ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:12:42.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.541+0000 7f329ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:42.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.541+0000 7f329ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:42.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.541+0000 7f329ffff640 10 monclient(hunting): my global_id is 4137 2024-03-29T11:12:42.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.541+0000 7f329ffff640 1 -- 172.21.15.121:0/319836133 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32a80b1da0 con 0x7f32a8145e60 2024-03-29T11:12:42.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.541+0000 7f329ffff640 1 -- 172.21.15.121:0/319836133 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3384966003 0 0) 0x7f3294003520 con 0x7f32a806af60 2024-03-29T11:12:42.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.541+0000 7f329ffff640 1 -- 172.21.15.121:0/319836133 >> v1:172.21.15.47:6790/0 conn(0x7f32a80af360 legacy=0x7f32a806c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:42.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.541+0000 7f329ffff640 1 -- 172.21.15.121:0/319836133 >> v1:172.21.15.47:6789/0 conn(0x7f32a8145e60 legacy=0x7f32a814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:42.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.541+0000 7f329ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:12:42.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.541+0000 7f329ffff640 1 monclient: found mon.noname-c 2024-03-29T11:12:42.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.541+0000 7f329ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:42.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.541+0000 7f329ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:12:42.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.541+0000 7f329ffff640 1 -- 172.21.15.121:0/319836133 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a80b2080 con 0x7f32a806af60 2024-03-29T11:12:42.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.541+0000 7f329ffff640 10 monclient: _finish_auth 0 2024-03-29T11:12:42.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.541+0000 7f329ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:12:42.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.541+0000 7f329ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:42.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.541+0000 7f329ffff640 1 -- 172.21.15.121:0/319836133 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f3298001690 con 0x7f32a806af60 2024-03-29T11:12:42.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.541+0000 7f329ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:42.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.541+0000 7f329ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:12:42.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.541+0000 7f329ffff640 10 monclient: dump: 2024-03-29T11:12:42.532 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:12:42.533 INFO:teuthology.orchestra.run.smithi121.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:42.533 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:42.533 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:42.533 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:42.533 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:42.533 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:42.533 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:42.533 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:42.533 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:42.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.541+0000 7f32af3ba640 5 monclient: authenticate success, global_id 4132 2024-03-29T11:12:42.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.541+0000 7f32af3ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:42.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.541+0000 7f329ffff640 1 -- 172.21.15.121:0/319836133 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3298001870 con 0x7f32a806af60 2024-03-29T11:12:42.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.541+0000 7f329ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:42.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.541+0000 7f329ffff640 1 -- 172.21.15.121:0/319836133 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f3298001c00 con 0x7f32a806af60 2024-03-29T11:12:42.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.541+0000 7f329ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:42.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.541+0000 7f329ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:42.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.541+0000 7f329ffff640 10 monclient: dump: 2024-03-29T11:12:42.534 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:12:42.534 INFO:teuthology.orchestra.run.smithi121.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:42.534 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:42.534 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:42.534 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:42.534 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:42.534 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:42.534 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:42.534 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:42.535 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:42.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.541+0000 7f32af3ba640 10 monclient: get_monmap_and_config success 2024-03-29T11:12:42.541 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.541+0000 7f32af3ba640 4 set_mon_vals no callback set 2024-03-29T11:12:42.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.542+0000 7f32af3ba640 10 monclient: shutdown 2024-03-29T11:12:42.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.542+0000 7f32af3ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:42.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.542+0000 7f32af3ba640 1 -- 172.21.15.121:0/319836133 >> v1:172.21.15.121:6789/0 conn(0x7f32a806af60 legacy=0x7f32a8145750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:42.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.542+0000 7f32af3ba640 1 -- 172.21.15.121:0/319836133 shutdown_connections 2024-03-29T11:12:42.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.542+0000 7f32af3ba640 1 -- 172.21.15.121:0/319836133 >> 172.21.15.121:0/319836133 conn(0x7f32a80b58f0 msgr2=0x7f32a80b5d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:42.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.542+0000 7f32af3ba640 1 -- 172.21.15.121:0/319836133 shutdown_connections 2024-03-29T11:12:42.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.542+0000 7f32af3ba640 1 -- 172.21.15.121:0/319836133 wait complete. 2024-03-29T11:12:42.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.543+0000 7f32af3ba640 10 monclient: build_initial_monmap 2024-03-29T11:12:42.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.543+0000 7f32af3ba640 10 monclient: monmap: 2024-03-29T11:12:42.542 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-03-29T11:12:42.543 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:12:42.543 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-03-29T11:12:42.543 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-03-29T11:12:42.543 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:42.543 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:42.543 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:12:42.543 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:12:42.543 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:12:42.543 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:42.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.543+0000 7f32af3ba640 1 librados: starting msgr at 2024-03-29T11:12:42.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.543+0000 7f32af3ba640 1 librados: starting objecter 2024-03-29T11:12:42.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.543+0000 7f32af3ba640 1 Processor -- start 2024-03-29T11:12:42.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.543+0000 7f32af3ba640 1 -- start start 2024-03-29T11:12:42.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.543+0000 7f32af3ba640 1 librados: setting wanted keys 2024-03-29T11:12:42.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.543+0000 7f32af3ba640 1 librados: calling monclient init 2024-03-29T11:12:42.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.543+0000 7f32af3ba640 10 monclient: init 2024-03-29T11:12:42.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.544+0000 7f32af3ba640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:42.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.544+0000 7f32af3ba640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:12:42.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.544+0000 7f32af3ba640 10 monclient(hunting): picked mon.noname-a con 0x7f32a806af60 addr v1:172.21.15.47:6789/0 2024-03-29T11:12:42.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.544+0000 7f32af3ba640 10 monclient(hunting): picked mon.noname-c con 0x7f32a80af360 addr v1:172.21.15.47:6790/0 2024-03-29T11:12:42.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.544+0000 7f32af3ba640 10 monclient(hunting): picked mon.noname-b con 0x7f32a8145e60 addr v1:172.21.15.121:6789/0 2024-03-29T11:12:42.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.544+0000 7f32af3ba640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32a811f0d0 con 0x7f32a806af60 2024-03-29T11:12:42.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.544+0000 7f32af3ba640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32a811fe40 con 0x7f32a8145e60 2024-03-29T11:12:42.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.544+0000 7f32af3ba640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32a81204c0 con 0x7f32a80af360 2024-03-29T11:12:42.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.544+0000 7f32af3ba640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:42.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.544+0000 7f32af3ba640 10 monclient(hunting): authenticate will time out at 816.710422s 2024-03-29T11:12:42.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.544+0000 7f32ad957640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f32a8145e60 0x7f32a81e2790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:54688/0 (socket says 172.21.15.121:54688) 2024-03-29T11:12:42.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.544+0000 7f32ad957640 1 -- 172.21.15.121:0/1651469217 learned_addr learned my addr 172.21.15.121:0/1651469217 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-03-29T11:12:42.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.544+0000 7f329dffb640 10 client.?.objecter ms_handle_connect 0x7f32a8145e60 2024-03-29T11:12:42.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.544+0000 7f329dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:42.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.544+0000 7f329dffb640 10 client.?.objecter ms_handle_connect 0x7f32a806af60 2024-03-29T11:12:42.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.544+0000 7f329dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:42.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.544+0000 7f329dffb640 10 client.?.objecter ms_handle_connect 0x7f32a80af360 2024-03-29T11:12:42.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.544+0000 7f329dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:42.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.544+0000 7f329dffb640 1 -- 172.21.15.121:0/1651469217 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1867229678 0 0) 0x7f32a811fe40 con 0x7f32a8145e60 2024-03-29T11:12:42.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.544+0000 7f329dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:12:42.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.544+0000 7f329dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:42.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.544+0000 7f329dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:42.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.544+0000 7f329dffb640 10 monclient(hunting): my global_id is 4135 2024-03-29T11:12:42.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.544+0000 7f329dffb640 1 -- 172.21.15.121:0/1651469217 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f328c004eb0 con 0x7f32a8145e60 2024-03-29T11:12:42.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.544+0000 7f329dffb640 1 -- 172.21.15.121:0/1651469217 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1264274505 0 0) 0x7f32a811f0d0 con 0x7f32a806af60 2024-03-29T11:12:42.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.544+0000 7f329dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:12:42.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.544+0000 7f329dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:42.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.544+0000 7f329dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:42.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.544+0000 7f329dffb640 10 monclient(hunting): my global_id is 4140 2024-03-29T11:12:42.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.544+0000 7f329dffb640 1 -- 172.21.15.121:0/1651469217 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32a811fe40 con 0x7f32a806af60 2024-03-29T11:12:42.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.544+0000 7f329dffb640 1 -- 172.21.15.121:0/1651469217 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1136638970 0 0) 0x7f32a81204c0 con 0x7f32a80af360 2024-03-29T11:12:42.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.544+0000 7f329dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:12:42.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.544+0000 7f329dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:42.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.544+0000 7f329dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:42.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.544+0000 7f329dffb640 10 monclient(hunting): my global_id is 4139 2024-03-29T11:12:42.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.544+0000 7f329dffb640 1 -- 172.21.15.121:0/1651469217 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32a811f0d0 con 0x7f32a80af360 2024-03-29T11:12:42.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.544+0000 7f329dffb640 1 -- 172.21.15.121:0/1651469217 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 927208868 0 0) 0x7f328c004eb0 con 0x7f32a8145e60 2024-03-29T11:12:42.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.544+0000 7f329dffb640 1 -- 172.21.15.121:0/1651469217 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f32a81204c0 con 0x7f32a8145e60 2024-03-29T11:12:42.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.544+0000 7f329dffb640 1 -- 172.21.15.121:0/1651469217 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f32a0001c70 con 0x7f32a8145e60 2024-03-29T11:12:42.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.544+0000 7f329dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:42.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.544+0000 7f329dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:12:42.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.544+0000 7f329dffb640 10 monclient(hunting): dump: 2024-03-29T11:12:42.547 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:12:42.547 INFO:teuthology.orchestra.run.smithi121.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:42.547 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:42.548 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:42.548 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:42.548 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:42.548 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:42.548 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:42.548 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:42.548 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:42.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.544+0000 7f329dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:12:42.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.544+0000 7f329dffb640 1 -- 172.21.15.121:0/1651469217 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2053661636 0 0) 0x7f32a811fe40 con 0x7f32a806af60 2024-03-29T11:12:42.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.544+0000 7f329dffb640 1 -- 172.21.15.121:0/1651469217 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f328c004eb0 con 0x7f32a806af60 2024-03-29T11:12:42.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.544+0000 7f329dffb640 1 -- 172.21.15.121:0/1651469217 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f32a40018b0 con 0x7f32a806af60 2024-03-29T11:12:42.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.544+0000 7f329dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:42.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.544+0000 7f329dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:12:42.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.544+0000 7f329dffb640 10 monclient(hunting): dump: 2024-03-29T11:12:42.550 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:12:42.550 INFO:teuthology.orchestra.run.smithi121.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:42.550 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:42.550 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:42.550 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:42.550 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:42.550 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:42.550 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:42.550 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:42.550 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:42.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.545+0000 7f329dffb640 1 -- 172.21.15.121:0/1651469217 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4202051389 0 0) 0x7f32a811f0d0 con 0x7f32a80af360 2024-03-29T11:12:42.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.545+0000 7f329dffb640 1 -- 172.21.15.121:0/1651469217 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f32a811fe40 con 0x7f32a80af360 2024-03-29T11:12:42.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.545+0000 7f329dffb640 1 -- 172.21.15.121:0/1651469217 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f32980019e0 con 0x7f32a80af360 2024-03-29T11:12:42.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.545+0000 7f329dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:42.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.545+0000 7f329dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:12:42.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.545+0000 7f329dffb640 10 monclient(hunting): dump: 2024-03-29T11:12:42.551 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:12:42.551 INFO:teuthology.orchestra.run.smithi121.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:42.551 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:42.551 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:42.551 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:42.551 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:42.551 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:42.551 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:42.552 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:42.552 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:42.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.545+0000 7f329dffb640 1 -- 172.21.15.121:0/1651469217 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1520172946 0 0) 0x7f32a81204c0 con 0x7f32a8145e60 2024-03-29T11:12:42.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.545+0000 7f329dffb640 1 -- 172.21.15.121:0/1651469217 >> v1:172.21.15.47:6790/0 conn(0x7f32a80af360 legacy=0x7f32a81da3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:42.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.545+0000 7f329dffb640 1 -- 172.21.15.121:0/1651469217 >> v1:172.21.15.47:6789/0 conn(0x7f32a806af60 legacy=0x7f32a813e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:42.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.545+0000 7f329dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:12:42.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.545+0000 7f329dffb640 1 monclient: found mon.b 2024-03-29T11:12:42.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.545+0000 7f329dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:42.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.545+0000 7f329dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:12:42.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.545+0000 7f329dffb640 1 -- 172.21.15.121:0/1651469217 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a8120650 con 0x7f32a8145e60 2024-03-29T11:12:42.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.545+0000 7f329dffb640 10 monclient: _finish_auth 0 2024-03-29T11:12:42.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.545+0000 7f329dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:12:42.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.545+0000 7f329dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:42.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.545+0000 7f32af3ba640 5 monclient: authenticate success, global_id 4135 2024-03-29T11:12:42.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.545+0000 7f32af3ba640 10 monclient: _renew_subs 2024-03-29T11:12:42.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.545+0000 7f32af3ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:12:42.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.545+0000 7f32af3ba640 1 -- 172.21.15.121:0/1651469217 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32a811f860 con 0x7f32a8145e60 2024-03-29T11:12:42.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.545+0000 7f329dffb640 1 -- 172.21.15.121:0/1651469217 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f32a0003990 con 0x7f32a8145e60 2024-03-29T11:12:42.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.545+0000 7f329dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:42.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.545+0000 7f329dffb640 1 -- 172.21.15.121:0/1651469217 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f32a0003da0 con 0x7f32a8145e60 2024-03-29T11:12:42.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.545+0000 7f329dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:42.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.545+0000 7f329dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:42.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.545+0000 7f329dffb640 10 monclient: dump: 2024-03-29T11:12:42.553 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:12:42.553 INFO:teuthology.orchestra.run.smithi121.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:42.553 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:42.553 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:42.553 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:42.553 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:42.554 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:42.554 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:42.554 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:42.554 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:42.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.545+0000 7f329dffb640 1 -- 172.21.15.121:0/1651469217 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f32a0004310 con 0x7f32a8145e60 2024-03-29T11:12:42.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.545+0000 7f32af3ba640 10 client.4135.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:12:42.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.545+0000 7f329f7fe640 4 set_mon_vals no callback set 2024-03-29T11:12:42.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.545+0000 7f32af3ba640 10 monclient: _renew_subs 2024-03-29T11:12:42.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.545+0000 7f32af3ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:12:42.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.545+0000 7f32af3ba640 1 -- 172.21.15.121:0/1651469217 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f32a813e9c0 con 0x7f32a8145e60 2024-03-29T11:12:42.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.545+0000 7f32af3ba640 1 librados: init done 2024-03-29T11:12:42.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.545+0000 7f329dffb640 1 -- 172.21.15.121:0/1651469217 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 308019933 0 0) 0x7f32a0006680 con 0x7f32a8145e60 2024-03-29T11:12:42.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.545+0000 7f329dffb640 10 client.4135.objecter ms_dispatch 0x7f32a8000b90 osd_map(6..6 src has 1..6) v4 2024-03-29T11:12:42.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.545+0000 7f329dffb640 3 client.4135.objecter handle_osd_map got epochs [6,6] > 0 2024-03-29T11:12:42.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.545+0000 7f329dffb640 3 client.4135.objecter handle_osd_map decoding full epoch 6 2024-03-29T11:12:42.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.545+0000 7f329dffb640 20 client.4135.objecter dump_active .. 0 homeless 2024-03-29T11:12:42.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.546+0000 7f32af3ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:42.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.546+0000 7f32af3ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:42.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.546+0000 7f32af3ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:12:42.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.546+0000 7f32af3ba640 1 -- 172.21.15.121:0/1651469217 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32a8067c00 con 0x7f32a8145e60 2024-03-29T11:12:42.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.548+0000 7f329dffb640 1 -- 172.21.15.121:0/1651469217 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f32a00047f0 con 0x7f32a8145e60 2024-03-29T11:12:42.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.548+0000 7f329dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:42.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.548+0000 7f329dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:12:42.556 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-29T11:12:42.565+0000 7f3465a3c040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:12:42.556 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-29T11:12:42.564+0000 7fcab480f040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:12:42.589 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-29T11:12:42.612+0000 7f3465a3c040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:12:42.596 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-29T11:12:42.618+0000 7fcab480f040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:12:42.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.619+0000 7f32af3ba640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "e1de122b-37a5-469f-bff2-bd60d934e4d2", "id": 4}] 2024-03-29T11:12:42.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.619+0000 7f32af3ba640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "e1de122b-37a5-469f-bff2-bd60d934e4d2", "id": 4}] 2024-03-29T11:12:42.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.619+0000 7f32af3ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:12:42.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.619+0000 7f32af3ba640 1 -- 172.21.15.121:0/1651469217 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd new", "uuid": "e1de122b-37a5-469f-bff2-bd60d934e4d2", "id": 4} v 0) v1 -- 0x7f32a811ffd0 con 0x7f32a8145e60 2024-03-29T11:12:42.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.623+0000 7f329dffb640 1 -- 172.21.15.121:0/1651469217 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e1de122b-37a5-469f-bff2-bd60d934e4d2", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 2302471812 0 1697012583) 0x7f32a0005c10 con 0x7f32a8145e60 2024-03-29T11:12:42.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.623+0000 7f329dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "e1de122b-37a5-469f-bff2-bd60d934e4d2", "id": 4}] 2024-03-29T11:12:42.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.623+0000 7f329dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:12:42.602 INFO:teuthology.orchestra.run.smithi121.stdout:4 2024-03-29T11:12:42.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.623+0000 7f32af3ba640 10 librados: watch_flush enter 2024-03-29T11:12:42.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.623+0000 7f32af3ba640 10 librados: watch_flush exit 2024-03-29T11:12:42.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.623+0000 7f32af3ba640 20 client.4135.objecter shutdown clearing up homeless session... 2024-03-29T11:12:42.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.623+0000 7f32af3ba640 10 client.4135.objecter successfully canceled tick 2024-03-29T11:12:42.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.623+0000 7f32af3ba640 10 monclient: shutdown 2024-03-29T11:12:42.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.623+0000 7f32af3ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:42.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.623+0000 7f32af3ba640 1 -- 172.21.15.121:0/1651469217 >> v1:172.21.15.121:6789/0 conn(0x7f32a8145e60 legacy=0x7f32a81e2790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:42.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.623+0000 7f32af3ba640 1 -- 172.21.15.121:0/1651469217 shutdown_connections 2024-03-29T11:12:42.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.623+0000 7f32af3ba640 1 -- 172.21.15.121:0/1651469217 >> 172.21.15.121:0/1651469217 conn(0x7f32a80b58f0 msgr2=0x7f32a80b47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:42.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.624+0000 7f32af3ba640 1 -- 172.21.15.121:0/1651469217 shutdown_connections 2024-03-29T11:12:42.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.624+0000 7f32af3ba640 1 -- 172.21.15.121:0/1651469217 wait complete. 2024-03-29T11:12:42.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.624+0000 7f32af3ba640 1 librados: shutdown 2024-03-29T11:12:42.612 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new 4ac1814f-2e59-4bce-a68b-83e395b1f724 5 2024-03-29T11:12:42.645 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-29T11:12:42.668+0000 7f3465a3c040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:12:42.646 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-29T11:12:42.668+0000 7fcab480f040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:12:42.694 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-29T11:12:42.716+0000 7f3465a3c040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:12:42.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.733+0000 7f7a1dbed640 10 monclient: get_monmap_and_config 2024-03-29T11:12:42.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.733+0000 7f7a1dbed640 10 monclient: build_initial_monmap 2024-03-29T11:12:42.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.733+0000 7f7a1dbed640 10 monclient: monmap: 2024-03-29T11:12:42.711 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-03-29T11:12:42.711 INFO:teuthology.orchestra.run.smithi121.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:42.711 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:12:42.734241+0000 2024-03-29T11:12:42.711 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:12:42.734241+0000 2024-03-29T11:12:42.712 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:42.712 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:42.712 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:12:42.712 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:12:42.712 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:12:42.713 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:42.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.734+0000 7f7a1dbed640 1 Processor -- start 2024-03-29T11:12:42.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.734+0000 7f7a1dbed640 1 -- start start 2024-03-29T11:12:42.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.734+0000 7f7a1dbed640 10 monclient: init 2024-03-29T11:12:42.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.734+0000 7f7a1dbed640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:42.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.734+0000 7f7a1dbed640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:12:42.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.734+0000 7f7a1dbed640 10 monclient(hunting): picked mon.noname-a con 0x7f7a1809ac40 addr v1:172.21.15.47:6789/0 2024-03-29T11:12:42.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.734+0000 7f7a1dbed640 10 monclient(hunting): picked mon.noname-b con 0x7f7a1809c870 addr v1:172.21.15.47:6790/0 2024-03-29T11:12:42.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.734+0000 7f7a1dbed640 10 monclient(hunting): picked mon.noname-c con 0x7f7a180b4360 addr v1:172.21.15.121:6789/0 2024-03-29T11:12:42.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.734+0000 7f7a1dbed640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a1809dec0 con 0x7f7a1809ac40 2024-03-29T11:12:42.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.734+0000 7f7a1dbed640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a1809da30 con 0x7f7a180b4360 2024-03-29T11:12:42.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.734+0000 7f7a1dbed640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a180b39c0 con 0x7f7a1809c870 2024-03-29T11:12:42.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.734+0000 7f7a1dbed640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:42.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.734+0000 7f7a1dbed640 10 monclient(hunting): authenticate will time out at 816.900779s 2024-03-29T11:12:42.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.734+0000 7f7a17fff640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f7a180b4360 0x7f7a1814d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:54690/0 (socket says 172.21.15.121:54690) 2024-03-29T11:12:42.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.734+0000 7f7a17fff640 1 -- 172.21.15.121:0/3859724003 learned_addr learned my addr 172.21.15.121:0/3859724003 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-03-29T11:12:42.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.735+0000 7f7a167fc640 1 -- 172.21.15.121:0/3859724003 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3211483299 0 0) 0x7f7a1809da30 con 0x7f7a180b4360 2024-03-29T11:12:42.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.735+0000 7f7a167fc640 10 monclient(hunting): _negotiate 2024-03-29T11:12:42.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.735+0000 7f7a167fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:42.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.735+0000 7f7a167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:42.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.735+0000 7f7a167fc640 10 monclient(hunting): my global_id is 4138 2024-03-29T11:12:42.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.735+0000 7f7a167fc640 1 -- 172.21.15.121:0/3859724003 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a00003520 con 0x7f7a180b4360 2024-03-29T11:12:42.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.735+0000 7f7a167fc640 1 -- 172.21.15.121:0/3859724003 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 476171750 0 0) 0x7f7a08001110 con 0x7f7a1809ac40 2024-03-29T11:12:42.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.735+0000 7f7a167fc640 10 monclient(hunting): _negotiate 2024-03-29T11:12:42.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.735+0000 7f7a167fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:42.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.735+0000 7f7a167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:42.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.735+0000 7f7a167fc640 10 monclient(hunting): my global_id is 4143 2024-03-29T11:12:42.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.735+0000 7f7a167fc640 1 -- 172.21.15.121:0/3859724003 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a1809da30 con 0x7f7a1809ac40 2024-03-29T11:12:42.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.735+0000 7f7a167fc640 1 -- 172.21.15.121:0/3859724003 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 511584157 0 0) 0x7f7a180b39c0 con 0x7f7a1809c870 2024-03-29T11:12:42.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.735+0000 7f7a167fc640 10 monclient(hunting): _negotiate 2024-03-29T11:12:42.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.735+0000 7f7a167fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:42.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.735+0000 7f7a167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:42.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.735+0000 7f7a167fc640 10 monclient(hunting): my global_id is 4142 2024-03-29T11:12:42.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.735+0000 7f7a167fc640 1 -- 172.21.15.121:0/3859724003 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a08001110 con 0x7f7a1809c870 2024-03-29T11:12:42.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.735+0000 7f7a167fc640 1 -- 172.21.15.121:0/3859724003 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2417937660 0 0) 0x7f7a00003520 con 0x7f7a180b4360 2024-03-29T11:12:42.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.735+0000 7f7a167fc640 1 -- 172.21.15.121:0/3859724003 >> v1:172.21.15.47:6790/0 conn(0x7f7a1809c870 legacy=0x7f7a1813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:42.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.735+0000 7f7a167fc640 1 -- 172.21.15.121:0/3859724003 >> v1:172.21.15.47:6789/0 conn(0x7f7a1809ac40 legacy=0x7f7a180b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:42.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.735+0000 7f7a167fc640 10 monclient: _finish_hunting 0 2024-03-29T11:12:42.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.735+0000 7f7a167fc640 1 monclient: found mon.noname-c 2024-03-29T11:12:42.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.735+0000 7f7a167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:42.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.735+0000 7f7a167fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:12:42.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.735+0000 7f7a167fc640 1 -- 172.21.15.121:0/3859724003 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a180b5160 con 0x7f7a180b4360 2024-03-29T11:12:42.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.735+0000 7f7a167fc640 10 monclient: _finish_auth 0 2024-03-29T11:12:42.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.735+0000 7f7a167fc640 10 monclient: _check_auth_tickets 2024-03-29T11:12:42.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.735+0000 7f7a167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:42.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.735+0000 7f7a1dbed640 5 monclient: authenticate success, global_id 4138 2024-03-29T11:12:42.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.735+0000 7f7a1dbed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:42.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.735+0000 7f7a167fc640 1 -- 172.21.15.121:0/3859724003 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f7a10001a10 con 0x7f7a180b4360 2024-03-29T11:12:42.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.735+0000 7f7a167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:42.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.735+0000 7f7a167fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:12:42.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.735+0000 7f7a167fc640 10 monclient: dump: 2024-03-29T11:12:42.728 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:12:42.728 INFO:teuthology.orchestra.run.smithi121.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:42.729 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:42.729 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:42.729 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:42.729 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:42.729 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:42.729 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:42.729 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:42.729 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:42.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.735+0000 7f7a1dbed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:42.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.735+0000 7f7a167fc640 1 -- 172.21.15.121:0/3859724003 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a10001bf0 con 0x7f7a180b4360 2024-03-29T11:12:42.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.735+0000 7f7a167fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:42.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.735+0000 7f7a167fc640 1 -- 172.21.15.121:0/3859724003 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f7a10002000 con 0x7f7a180b4360 2024-03-29T11:12:42.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.735+0000 7f7a167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:42.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.735+0000 7f7a167fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:42.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.735+0000 7f7a167fc640 10 monclient: dump: 2024-03-29T11:12:42.730 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:12:42.730 INFO:teuthology.orchestra.run.smithi121.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:42.730 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:42.730 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:42.730 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:42.730 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:42.730 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:42.730 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:42.730 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:42.730 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:42.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.735+0000 7f7a1dbed640 10 monclient: get_monmap_and_config success 2024-03-29T11:12:42.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.735+0000 7f7a1dbed640 4 set_mon_vals no callback set 2024-03-29T11:12:42.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.736+0000 7f7a1dbed640 10 monclient: shutdown 2024-03-29T11:12:42.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.736+0000 7f7a1dbed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:42.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.736+0000 7f7a1dbed640 1 -- 172.21.15.121:0/3859724003 >> v1:172.21.15.121:6789/0 conn(0x7f7a180b4360 legacy=0x7f7a1814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:42.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.736+0000 7f7a1dbed640 1 -- 172.21.15.121:0/3859724003 shutdown_connections 2024-03-29T11:12:42.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.736+0000 7f7a1dbed640 1 -- 172.21.15.121:0/3859724003 >> 172.21.15.121:0/3859724003 conn(0x7f7a18066b00 msgr2=0x7f7a18066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:42.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.736+0000 7f7a1dbed640 1 -- 172.21.15.121:0/3859724003 shutdown_connections 2024-03-29T11:12:42.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.736+0000 7f7a1dbed640 1 -- 172.21.15.121:0/3859724003 wait complete. 2024-03-29T11:12:42.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.736+0000 7f7a1dbed640 10 monclient: build_initial_monmap 2024-03-29T11:12:42.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.736+0000 7f7a1dbed640 10 monclient: monmap: 2024-03-29T11:12:42.731 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-03-29T11:12:42.731 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:12:42.731 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-03-29T11:12:42.731 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-03-29T11:12:42.731 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:42.731 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:42.731 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:12:42.732 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:12:42.732 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:12:42.732 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:42.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.737+0000 7f7a1dbed640 1 librados: starting msgr at 2024-03-29T11:12:42.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.737+0000 7f7a1dbed640 1 librados: starting objecter 2024-03-29T11:12:42.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.737+0000 7f7a1dbed640 1 Processor -- start 2024-03-29T11:12:42.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.737+0000 7f7a1dbed640 1 -- start start 2024-03-29T11:12:42.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.737+0000 7f7a1dbed640 1 librados: setting wanted keys 2024-03-29T11:12:42.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.737+0000 7f7a1dbed640 1 librados: calling monclient init 2024-03-29T11:12:42.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.737+0000 7f7a1dbed640 10 monclient: init 2024-03-29T11:12:42.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f7a1dbed640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:42.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f7a1dbed640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:12:42.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f7a1dbed640 10 monclient(hunting): picked mon.noname-b con 0x7f7a1809ac40 addr v1:172.21.15.121:6789/0 2024-03-29T11:12:42.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f7a1dbed640 10 monclient(hunting): picked mon.noname-a con 0x7f7a18141910 addr v1:172.21.15.47:6789/0 2024-03-29T11:12:42.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f7a1dbed640 10 monclient(hunting): picked mon.noname-c con 0x7f7a1813ffa0 addr v1:172.21.15.47:6790/0 2024-03-29T11:12:42.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f7a1dbed640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a18139b60 con 0x7f7a18141910 2024-03-29T11:12:42.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f7a1dbed640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a181d6f10 con 0x7f7a1809ac40 2024-03-29T11:12:42.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f7a1dbed640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a180afba0 con 0x7f7a1813ffa0 2024-03-29T11:12:42.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f7a1dbed640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:42.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f7a1dbed640 10 monclient(hunting): authenticate will time out at 816.904231s 2024-03-29T11:12:42.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f7a177fe640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f7a1809ac40 0x7f7a18141200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:54706/0 (socket says 172.21.15.121:54706) 2024-03-29T11:12:42.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f7a177fe640 1 -- 172.21.15.121:0/3444354247 learned_addr learned my addr 172.21.15.121:0/3444354247 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-03-29T11:12:42.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f79fffff640 10 client.?.objecter ms_handle_connect 0x7f7a1809ac40 2024-03-29T11:12:42.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f79fffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:42.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f79fffff640 10 client.?.objecter ms_handle_connect 0x7f7a1813ffa0 2024-03-29T11:12:42.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f79fffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:42.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f79fffff640 10 client.?.objecter ms_handle_connect 0x7f7a18141910 2024-03-29T11:12:42.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f79fffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:42.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f79fffff640 1 -- 172.21.15.121:0/3444354247 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 370222280 0 0) 0x7f7a181d6f10 con 0x7f7a1809ac40 2024-03-29T11:12:42.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f79fffff640 10 monclient(hunting): _negotiate 2024-03-29T11:12:42.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f79fffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:42.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f79fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:42.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f79fffff640 10 monclient(hunting): my global_id is 4141 2024-03-29T11:12:42.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f79fffff640 1 -- 172.21.15.121:0/3444354247 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a04004eb0 con 0x7f7a1809ac40 2024-03-29T11:12:42.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f79fffff640 1 -- 172.21.15.121:0/3444354247 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1991985400 0 0) 0x7f7a180afba0 con 0x7f7a1813ffa0 2024-03-29T11:12:42.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f79fffff640 10 monclient(hunting): _negotiate 2024-03-29T11:12:42.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f79fffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:42.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f79fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:42.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f79fffff640 10 monclient(hunting): my global_id is 4145 2024-03-29T11:12:42.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f79fffff640 1 -- 172.21.15.121:0/3444354247 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a181d6f10 con 0x7f7a1813ffa0 2024-03-29T11:12:42.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f79fffff640 1 -- 172.21.15.121:0/3444354247 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 869883941 0 0) 0x7f7a08001110 con 0x7f7a18141910 2024-03-29T11:12:42.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f79fffff640 10 monclient(hunting): _negotiate 2024-03-29T11:12:42.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f79fffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:42.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f79fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:42.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f79fffff640 10 monclient(hunting): my global_id is 4146 2024-03-29T11:12:42.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f79fffff640 1 -- 172.21.15.121:0/3444354247 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a180afba0 con 0x7f7a18141910 2024-03-29T11:12:42.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f79fffff640 1 -- 172.21.15.121:0/3444354247 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4065190748 0 0) 0x7f7a04004eb0 con 0x7f7a1809ac40 2024-03-29T11:12:42.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f79fffff640 1 -- 172.21.15.121:0/3444354247 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a08001110 con 0x7f7a1809ac40 2024-03-29T11:12:42.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f79fffff640 1 -- 172.21.15.121:0/3444354247 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f7a080019a0 con 0x7f7a1809ac40 2024-03-29T11:12:42.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f79fffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:42.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f79fffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:12:42.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f79fffff640 10 monclient(hunting): dump: 2024-03-29T11:12:42.735 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:12:42.735 INFO:teuthology.orchestra.run.smithi121.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:42.735 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:42.735 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:42.735 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:42.735 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:42.736 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:42.736 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:42.736 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:42.736 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:42.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f79fffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:12:42.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f79fffff640 1 -- 172.21.15.121:0/3444354247 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2814347999 0 0) 0x7f7a181d6f10 con 0x7f7a1813ffa0 2024-03-29T11:12:42.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f79fffff640 1 -- 172.21.15.121:0/3444354247 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a04004eb0 con 0x7f7a1813ffa0 2024-03-29T11:12:42.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f79fffff640 1 -- 172.21.15.121:0/3444354247 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 201469467 0 0) 0x7f7a180afba0 con 0x7f7a18141910 2024-03-29T11:12:42.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f79fffff640 1 -- 172.21.15.121:0/3444354247 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a181d6f10 con 0x7f7a18141910 2024-03-29T11:12:42.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f79fffff640 1 -- 172.21.15.121:0/3444354247 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f7a10001f00 con 0x7f7a1813ffa0 2024-03-29T11:12:42.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f79fffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:42.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f79fffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:12:42.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f79fffff640 10 monclient(hunting): dump: 2024-03-29T11:12:42.737 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:12:42.737 INFO:teuthology.orchestra.run.smithi121.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:42.737 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:42.737 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:42.737 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:42.737 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:42.737 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:42.737 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:42.737 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:42.737 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:42.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f79fffff640 1 -- 172.21.15.121:0/3444354247 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3643733117 0 0) 0x7f7a08001110 con 0x7f7a1809ac40 2024-03-29T11:12:42.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f79fffff640 1 -- 172.21.15.121:0/3444354247 >> v1:172.21.15.47:6790/0 conn(0x7f7a1813ffa0 legacy=0x7f7a18138e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:42.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f79fffff640 1 -- 172.21.15.121:0/3444354247 >> v1:172.21.15.47:6789/0 conn(0x7f7a18141910 legacy=0x7f7a181d73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:42.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f79fffff640 10 monclient: _finish_hunting 0 2024-03-29T11:12:42.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f79fffff640 1 monclient: found mon.b 2024-03-29T11:12:42.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f79fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:42.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f79fffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:12:42.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f79fffff640 1 -- 172.21.15.121:0/3444354247 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a18139520 con 0x7f7a1809ac40 2024-03-29T11:12:42.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f79fffff640 10 monclient: _finish_auth 0 2024-03-29T11:12:42.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f79fffff640 10 monclient: _check_auth_tickets 2024-03-29T11:12:42.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f79fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:42.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f7a1dbed640 5 monclient: authenticate success, global_id 4141 2024-03-29T11:12:42.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f7a1dbed640 10 monclient: _renew_subs 2024-03-29T11:12:42.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f7a1dbed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:12:42.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f7a1dbed640 1 -- 172.21.15.121:0/3444354247 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a18139710 con 0x7f7a1809ac40 2024-03-29T11:12:42.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f7a1dbed640 10 client.4141.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:12:42.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f7a1dbed640 10 monclient: _renew_subs 2024-03-29T11:12:42.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f7a1dbed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:12:42.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f7a1dbed640 1 -- 172.21.15.121:0/3444354247 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a181d7c90 con 0x7f7a1809ac40 2024-03-29T11:12:42.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.738+0000 7f7a1dbed640 1 librados: init done 2024-03-29T11:12:42.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.739+0000 7f79fffff640 1 -- 172.21.15.121:0/3444354247 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a08003610 con 0x7f7a1809ac40 2024-03-29T11:12:42.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.739+0000 7f79fffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:42.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.739+0000 7f79fffff640 1 -- 172.21.15.121:0/3444354247 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f7a08003920 con 0x7f7a1809ac40 2024-03-29T11:12:42.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.739+0000 7f79fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:42.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.739+0000 7f79fffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:42.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.739+0000 7f7a15ffb640 4 set_mon_vals no callback set 2024-03-29T11:12:42.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.739+0000 7f79fffff640 10 monclient: dump: 2024-03-29T11:12:42.740 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:12:42.740 INFO:teuthology.orchestra.run.smithi121.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:42.740 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:42.740 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:42.740 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:42.740 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:42.740 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:42.740 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:42.740 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:42.741 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:42.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.739+0000 7f79fffff640 1 -- 172.21.15.121:0/3444354247 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f7a08003ed0 con 0x7f7a1809ac40 2024-03-29T11:12:42.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.739+0000 7f79fffff640 1 -- 172.21.15.121:0/3444354247 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 3640318204 0 0) 0x7f7a08006780 con 0x7f7a1809ac40 2024-03-29T11:12:42.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.739+0000 7f79fffff640 10 client.4141.objecter ms_dispatch 0x7f7a18000b90 osd_map(7..7 src has 1..7) v4 2024-03-29T11:12:42.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.739+0000 7f79fffff640 3 client.4141.objecter handle_osd_map got epochs [7,7] > 0 2024-03-29T11:12:42.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.739+0000 7f79fffff640 3 client.4141.objecter handle_osd_map decoding full epoch 7 2024-03-29T11:12:42.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.739+0000 7f79fffff640 20 client.4141.objecter dump_active .. 0 homeless 2024-03-29T11:12:42.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.739+0000 7f7a1dbed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:42.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.739+0000 7f7a1dbed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:42.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.739+0000 7f7a1dbed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:12:42.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.739+0000 7f7a1dbed640 1 -- 172.21.15.121:0/3444354247 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79f8005080 con 0x7f7a1809ac40 2024-03-29T11:12:42.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.741+0000 7f79fffff640 1 -- 172.21.15.121:0/3444354247 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f7a08004310 con 0x7f7a1809ac40 2024-03-29T11:12:42.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.741+0000 7f79fffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:42.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.741+0000 7f79fffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:12:42.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.813+0000 7f7a1dbed640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "4ac1814f-2e59-4bce-a68b-83e395b1f724", "id": 5}] 2024-03-29T11:12:42.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.813+0000 7f7a1dbed640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "4ac1814f-2e59-4bce-a68b-83e395b1f724", "id": 5}] 2024-03-29T11:12:42.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.813+0000 7f7a1dbed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:12:42.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.813+0000 7f7a1dbed640 1 -- 172.21.15.121:0/3444354247 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd new", "uuid": "4ac1814f-2e59-4bce-a68b-83e395b1f724", "id": 5} v 0) v1 -- 0x7f79f80058b0 con 0x7f7a1809ac40 2024-03-29T11:12:42.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.817+0000 7f79fffff640 1 -- 172.21.15.121:0/3444354247 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4ac1814f-2e59-4bce-a68b-83e395b1f724", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 2388043536 0 1988412176) 0x7f7a080062d0 con 0x7f7a1809ac40 2024-03-29T11:12:42.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.817+0000 7f79fffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "4ac1814f-2e59-4bce-a68b-83e395b1f724", "id": 5}] 2024-03-29T11:12:42.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.817+0000 7f79fffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:12:42.795 INFO:teuthology.orchestra.run.smithi121.stdout:5 2024-03-29T11:12:42.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.818+0000 7f7a1dbed640 10 librados: watch_flush enter 2024-03-29T11:12:42.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.818+0000 7f7a1dbed640 10 librados: watch_flush exit 2024-03-29T11:12:42.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.818+0000 7f7a1dbed640 20 client.4141.objecter shutdown clearing up homeless session... 2024-03-29T11:12:42.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.818+0000 7f7a1dbed640 10 client.4141.objecter successfully canceled tick 2024-03-29T11:12:42.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.818+0000 7f7a1dbed640 10 monclient: shutdown 2024-03-29T11:12:42.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.818+0000 7f7a1dbed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:42.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.818+0000 7f7a1dbed640 1 -- 172.21.15.121:0/3444354247 >> v1:172.21.15.121:6789/0 conn(0x7f7a1809ac40 legacy=0x7f7a18141200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:42.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.818+0000 7f7a1dbed640 1 -- 172.21.15.121:0/3444354247 shutdown_connections 2024-03-29T11:12:42.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.818+0000 7f7a1dbed640 1 -- 172.21.15.121:0/3444354247 >> 172.21.15.121:0/3444354247 conn(0x7f7a18066b00 msgr2=0x7f7a1813c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:42.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.818+0000 7f7a1dbed640 1 -- 172.21.15.121:0/3444354247 shutdown_connections 2024-03-29T11:12:42.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.818+0000 7f7a1dbed640 1 -- 172.21.15.121:0/3444354247 wait complete. 2024-03-29T11:12:42.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.818+0000 7f7a1dbed640 1 librados: shutdown 2024-03-29T11:12:42.806 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new c06ac13d-25ef-476f-8acf-0f90ad4b2a00 6 2024-03-29T11:12:42.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.969+0000 7f4969a0d640 10 monclient: get_monmap_and_config 2024-03-29T11:12:42.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.969+0000 7f4969a0d640 10 monclient: build_initial_monmap 2024-03-29T11:12:42.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.969+0000 7f4969a0d640 10 monclient: monmap: 2024-03-29T11:12:42.947 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-03-29T11:12:42.947 INFO:teuthology.orchestra.run.smithi121.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:42.947 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:12:42.970432+0000 2024-03-29T11:12:42.948 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:12:42.970432+0000 2024-03-29T11:12:42.948 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:42.948 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:42.948 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:12:42.948 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:12:42.948 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:12:42.948 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:42.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.970+0000 7f4969a0d640 1 Processor -- start 2024-03-29T11:12:42.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.970+0000 7f4969a0d640 1 -- start start 2024-03-29T11:12:42.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.970+0000 7f4969a0d640 10 monclient: init 2024-03-29T11:12:42.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.970+0000 7f4969a0d640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:42.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.970+0000 7f4969a0d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:12:42.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.970+0000 7f4969a0d640 10 monclient(hunting): picked mon.noname-a con 0x7f49640b8240 addr v1:172.21.15.47:6789/0 2024-03-29T11:12:42.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.970+0000 7f4969a0d640 10 monclient(hunting): picked mon.noname-b con 0x7f49640b1f60 addr v1:172.21.15.47:6790/0 2024-03-29T11:12:42.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.970+0000 7f4969a0d640 10 monclient(hunting): picked mon.noname-c con 0x7f496409c8c0 addr v1:172.21.15.121:6789/0 2024-03-29T11:12:42.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.970+0000 7f4969a0d640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f496409bd80 con 0x7f49640b8240 2024-03-29T11:12:42.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.970+0000 7f4969a0d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f496409b960 con 0x7f496409c8c0 2024-03-29T11:12:42.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.970+0000 7f4969a0d640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49640b2ec0 con 0x7f49640b1f60 2024-03-29T11:12:42.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.970+0000 7f4969a0d640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:42.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.970+0000 7f4969a0d640 10 monclient(hunting): authenticate will time out at 817.136960s 2024-03-29T11:12:42.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.970+0000 7f49637fe640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f496409c8c0 0x7f496414d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:54710/0 (socket says 172.21.15.121:54710) 2024-03-29T11:12:42.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.970+0000 7f49637fe640 1 -- 172.21.15.121:0/3733758875 learned_addr learned my addr 172.21.15.121:0/3733758875 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-03-29T11:12:42.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.971+0000 7f4961ffb640 1 -- 172.21.15.121:0/3733758875 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2264526378 0 0) 0x7f496409b960 con 0x7f496409c8c0 2024-03-29T11:12:42.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.971+0000 7f4961ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:12:42.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.971+0000 7f4961ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:42.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.971+0000 7f4961ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:42.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.971+0000 7f4961ffb640 10 monclient(hunting): my global_id is 4144 2024-03-29T11:12:42.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.971+0000 7f4961ffb640 1 -- 172.21.15.121:0/3733758875 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f494c003520 con 0x7f496409c8c0 2024-03-29T11:12:42.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.971+0000 7f4961ffb640 1 -- 172.21.15.121:0/3733758875 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 337914892 0 0) 0x7f49640b2ec0 con 0x7f49640b1f60 2024-03-29T11:12:42.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.971+0000 7f4961ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:12:42.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.971+0000 7f4961ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:42.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.971+0000 7f4961ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:42.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.971+0000 7f4961ffb640 10 monclient(hunting): my global_id is 4148 2024-03-29T11:12:42.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.971+0000 7f4961ffb640 1 -- 172.21.15.121:0/3733758875 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f496409b960 con 0x7f49640b1f60 2024-03-29T11:12:42.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.971+0000 7f4961ffb640 1 -- 172.21.15.121:0/3733758875 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 758336367 0 0) 0x7f496409bd80 con 0x7f49640b8240 2024-03-29T11:12:42.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.971+0000 7f4961ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:12:42.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.971+0000 7f4961ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:42.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.971+0000 7f4961ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:42.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.971+0000 7f4961ffb640 10 monclient(hunting): my global_id is 4149 2024-03-29T11:12:42.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.971+0000 7f4961ffb640 1 -- 172.21.15.121:0/3733758875 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49640b2ec0 con 0x7f49640b8240 2024-03-29T11:12:42.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.971+0000 7f4961ffb640 1 -- 172.21.15.121:0/3733758875 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 834008999 0 0) 0x7f494c003520 con 0x7f496409c8c0 2024-03-29T11:12:42.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.971+0000 7f4961ffb640 1 -- 172.21.15.121:0/3733758875 >> v1:172.21.15.47:6790/0 conn(0x7f49640b1f60 legacy=0x7f496409c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:42.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.971+0000 7f4961ffb640 1 -- 172.21.15.121:0/3733758875 >> v1:172.21.15.47:6789/0 conn(0x7f49640b8240 legacy=0x7f496413f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:42.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.971+0000 7f4961ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:12:42.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.971+0000 7f4961ffb640 1 monclient: found mon.noname-c 2024-03-29T11:12:42.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.971+0000 7f4961ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:42.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.971+0000 7f4961ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:12:42.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.971+0000 7f4961ffb640 1 -- 172.21.15.121:0/3733758875 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f496413fb60 con 0x7f496409c8c0 2024-03-29T11:12:42.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.971+0000 7f4961ffb640 10 monclient: _finish_auth 0 2024-03-29T11:12:42.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.971+0000 7f4961ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:12:42.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.971+0000 7f4961ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:42.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.971+0000 7f4961ffb640 1 -- 172.21.15.121:0/3733758875 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f495c001a10 con 0x7f496409c8c0 2024-03-29T11:12:42.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.971+0000 7f4961ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:42.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.971+0000 7f4961ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:12:42.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.971+0000 7f4961ffb640 10 monclient: dump: 2024-03-29T11:12:42.963 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:12:42.963 INFO:teuthology.orchestra.run.smithi121.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:42.963 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:42.963 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:42.963 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:42.963 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:42.963 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:42.963 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:42.963 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:42.964 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:42.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.971+0000 7f4969a0d640 5 monclient: authenticate success, global_id 4144 2024-03-29T11:12:42.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.971+0000 7f4969a0d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:42.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.971+0000 7f4961ffb640 1 -- 172.21.15.121:0/3733758875 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f495c001bf0 con 0x7f496409c8c0 2024-03-29T11:12:42.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.971+0000 7f4961ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:42.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.971+0000 7f4969a0d640 10 monclient: get_monmap_and_config success 2024-03-29T11:12:42.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.971+0000 7f4961ffb640 1 -- 172.21.15.121:0/3733758875 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f495c002000 con 0x7f496409c8c0 2024-03-29T11:12:42.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.971+0000 7f4961ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:42.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.971+0000 7f4969a0d640 4 set_mon_vals no callback set 2024-03-29T11:12:42.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.971+0000 7f4961ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:42.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.971+0000 7f4961ffb640 10 monclient: dump: 2024-03-29T11:12:42.964 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:12:42.965 INFO:teuthology.orchestra.run.smithi121.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:42.965 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:42.965 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:42.965 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:42.965 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:42.965 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:42.965 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:42.965 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:42.965 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:42.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.972+0000 7f4969a0d640 10 monclient: shutdown 2024-03-29T11:12:42.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.972+0000 7f4969a0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:42.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.972+0000 7f4969a0d640 1 -- 172.21.15.121:0/3733758875 >> v1:172.21.15.121:6789/0 conn(0x7f496409c8c0 legacy=0x7f496414d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:42.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.972+0000 7f4969a0d640 1 -- 172.21.15.121:0/3733758875 shutdown_connections 2024-03-29T11:12:42.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.972+0000 7f4969a0d640 1 -- 172.21.15.121:0/3733758875 >> 172.21.15.121:0/3733758875 conn(0x7f4964066b00 msgr2=0x7f4964066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:42.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.972+0000 7f4969a0d640 1 -- 172.21.15.121:0/3733758875 shutdown_connections 2024-03-29T11:12:42.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.972+0000 7f4969a0d640 1 -- 172.21.15.121:0/3733758875 wait complete. 2024-03-29T11:12:42.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.972+0000 7f4969a0d640 10 monclient: build_initial_monmap 2024-03-29T11:12:42.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.972+0000 7f4969a0d640 10 monclient: monmap: 2024-03-29T11:12:42.966 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-03-29T11:12:42.966 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:12:42.966 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-03-29T11:12:42.966 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-03-29T11:12:42.966 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:42.966 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:42.966 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:12:42.966 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:12:42.966 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:12:42.967 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:42.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.973+0000 7f4969a0d640 1 librados: starting msgr at 2024-03-29T11:12:42.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.973+0000 7f4969a0d640 1 librados: starting objecter 2024-03-29T11:12:42.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.973+0000 7f4969a0d640 1 Processor -- start 2024-03-29T11:12:42.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.973+0000 7f4969a0d640 1 -- start start 2024-03-29T11:12:42.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.973+0000 7f4969a0d640 1 librados: setting wanted keys 2024-03-29T11:12:42.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.973+0000 7f4969a0d640 1 librados: calling monclient init 2024-03-29T11:12:42.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.973+0000 7f4969a0d640 10 monclient: init 2024-03-29T11:12:42.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.973+0000 7f4969a0d640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:42.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f4969a0d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:12:42.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f4969a0d640 10 monclient(hunting): picked mon.noname-a con 0x7f496409c8c0 addr v1:172.21.15.47:6789/0 2024-03-29T11:12:42.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f4969a0d640 10 monclient(hunting): picked mon.noname-b con 0x7f49640b8240 addr v1:172.21.15.121:6789/0 2024-03-29T11:12:42.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f4969a0d640 10 monclient(hunting): picked mon.noname-c con 0x7f4964141c90 addr v1:172.21.15.47:6790/0 2024-03-29T11:12:42.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f4969a0d640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4964133cc0 con 0x7f496409c8c0 2024-03-29T11:12:42.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f4969a0d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4964133fa0 con 0x7f49640b8240 2024-03-29T11:12:42.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f4969a0d640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4964134280 con 0x7f4964141c90 2024-03-29T11:12:42.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f4969a0d640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:42.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f4969a0d640 10 monclient(hunting): authenticate will time out at 817.140215s 2024-03-29T11:12:42.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f49627fc640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f49640b8240 0x7f4964131850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:54718/0 (socket says 172.21.15.121:54718) 2024-03-29T11:12:42.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f49627fc640 1 -- 172.21.15.121:0/2722039034 learned_addr learned my addr 172.21.15.121:0/2722039034 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-03-29T11:12:42.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f494bfff640 10 client.?.objecter ms_handle_connect 0x7f49640b8240 2024-03-29T11:12:42.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f494bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:42.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f494bfff640 10 client.?.objecter ms_handle_connect 0x7f496409c8c0 2024-03-29T11:12:42.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f494bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:42.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f494bfff640 10 client.?.objecter ms_handle_connect 0x7f4964141c90 2024-03-29T11:12:42.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f494bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:42.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f494bfff640 1 -- 172.21.15.121:0/2722039034 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2351494864 0 0) 0x7f4964133fa0 con 0x7f49640b8240 2024-03-29T11:12:42.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f494bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:12:42.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f494bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:42.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f494bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:42.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f494bfff640 10 monclient(hunting): my global_id is 4147 2024-03-29T11:12:42.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f494bfff640 1 -- 172.21.15.121:0/2722039034 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4950004eb0 con 0x7f49640b8240 2024-03-29T11:12:42.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f494bfff640 1 -- 172.21.15.121:0/2722039034 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3837138843 0 0) 0x7f4964133cc0 con 0x7f496409c8c0 2024-03-29T11:12:42.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f494bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:12:42.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f494bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:42.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f494bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:42.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f494bfff640 10 monclient(hunting): my global_id is 4152 2024-03-29T11:12:42.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f494bfff640 1 -- 172.21.15.121:0/2722039034 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4964133fa0 con 0x7f496409c8c0 2024-03-29T11:12:42.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f494bfff640 1 -- 172.21.15.121:0/2722039034 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4280124678 0 0) 0x7f4964134280 con 0x7f4964141c90 2024-03-29T11:12:42.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f494bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:12:42.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f494bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:42.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f494bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:42.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f494bfff640 10 monclient(hunting): my global_id is 4151 2024-03-29T11:12:42.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f494bfff640 1 -- 172.21.15.121:0/2722039034 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4964133cc0 con 0x7f4964141c90 2024-03-29T11:12:42.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f494bfff640 1 -- 172.21.15.121:0/2722039034 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3045009708 0 0) 0x7f4964133fa0 con 0x7f496409c8c0 2024-03-29T11:12:42.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f494bfff640 1 -- 172.21.15.121:0/2722039034 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4964134280 con 0x7f496409c8c0 2024-03-29T11:12:42.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f494bfff640 1 -- 172.21.15.121:0/2722039034 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f49540017d0 con 0x7f496409c8c0 2024-03-29T11:12:42.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f494bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:42.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f494bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:12:42.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f494bfff640 10 monclient(hunting): dump: 2024-03-29T11:12:42.970 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:12:42.970 INFO:teuthology.orchestra.run.smithi121.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:42.971 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:42.971 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:42.971 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:42.971 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:42.971 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:42.971 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:42.971 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:42.971 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:42.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f494bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:12:42.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f494bfff640 1 -- 172.21.15.121:0/2722039034 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 346677081 0 0) 0x7f4950004eb0 con 0x7f49640b8240 2024-03-29T11:12:42.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f494bfff640 1 -- 172.21.15.121:0/2722039034 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4964133fa0 con 0x7f49640b8240 2024-03-29T11:12:42.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f494bfff640 1 -- 172.21.15.121:0/2722039034 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 750537201 0 0) 0x7f4964133cc0 con 0x7f4964141c90 2024-03-29T11:12:42.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f494bfff640 1 -- 172.21.15.121:0/2722039034 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4950004eb0 con 0x7f4964141c90 2024-03-29T11:12:42.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f494bfff640 1 -- 172.21.15.121:0/2722039034 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f49580018f0 con 0x7f49640b8240 2024-03-29T11:12:42.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f494bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:42.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f494bfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:42.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f494bfff640 10 monclient(hunting): dump: 2024-03-29T11:12:42.972 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:12:42.972 INFO:teuthology.orchestra.run.smithi121.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:42.972 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:42.972 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:42.973 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:42.973 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:42.973 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:42.973 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:42.973 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:42.973 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:42.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f494bfff640 1 -- 172.21.15.121:0/2722039034 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f495c001d60 con 0x7f4964141c90 2024-03-29T11:12:42.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f494bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:42.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f494bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:12:42.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f494bfff640 10 monclient(hunting): dump: 2024-03-29T11:12:42.973 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:12:42.973 INFO:teuthology.orchestra.run.smithi121.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:42.974 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:42.974 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:42.974 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:42.974 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:42.974 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:42.974 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:42.974 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:42.974 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:42.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f494bfff640 1 -- 172.21.15.121:0/2722039034 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2825824498 0 0) 0x7f4964134280 con 0x7f496409c8c0 2024-03-29T11:12:42.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f494bfff640 1 -- 172.21.15.121:0/2722039034 >> v1:172.21.15.47:6790/0 conn(0x7f4964141c90 legacy=0x7f49641e2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:42.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f494bfff640 1 -- 172.21.15.121:0/2722039034 >> v1:172.21.15.121:6789/0 conn(0x7f49640b8240 legacy=0x7f4964131850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:42.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f494bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:12:42.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f494bfff640 1 monclient: found mon.a 2024-03-29T11:12:42.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f494bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:42.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f494bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:12:42.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f494bfff640 1 -- 172.21.15.121:0/2722039034 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4964142dc0 con 0x7f496409c8c0 2024-03-29T11:12:42.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f494bfff640 10 monclient: _finish_auth 0 2024-03-29T11:12:42.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f494bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:12:42.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f494bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:42.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f4969a0d640 5 monclient: authenticate success, global_id 4152 2024-03-29T11:12:42.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f4969a0d640 10 monclient: _renew_subs 2024-03-29T11:12:42.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f4969a0d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:12:42.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.974+0000 7f4969a0d640 1 -- 172.21.15.121:0/2722039034 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4964132ff0 con 0x7f496409c8c0 2024-03-29T11:12:42.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.975+0000 7f4969a0d640 10 client.4152.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:12:42.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.975+0000 7f4969a0d640 10 monclient: _renew_subs 2024-03-29T11:12:42.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.975+0000 7f4969a0d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:12:42.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.975+0000 7f4969a0d640 1 -- 172.21.15.121:0/2722039034 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4964135890 con 0x7f496409c8c0 2024-03-29T11:12:42.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.975+0000 7f4969a0d640 1 librados: init done 2024-03-29T11:12:42.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.975+0000 7f494bfff640 1 -- 172.21.15.121:0/2722039034 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4954001d60 con 0x7f496409c8c0 2024-03-29T11:12:42.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.975+0000 7f494bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:42.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.975+0000 7f494bfff640 1 -- 172.21.15.121:0/2722039034 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f49540037f0 con 0x7f496409c8c0 2024-03-29T11:12:42.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.975+0000 7f494bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:42.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.975+0000 7f494bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:12:42.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.975+0000 7f494bfff640 10 monclient: dump: 2024-03-29T11:12:42.976 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:12:42.976 INFO:teuthology.orchestra.run.smithi121.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:42.976 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:42.976 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:42.976 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:42.976 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:42.976 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:42.976 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:42.977 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:42.977 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:42.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.975+0000 7f49617fa640 4 set_mon_vals no callback set 2024-03-29T11:12:42.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.975+0000 7f494bfff640 1 -- 172.21.15.121:0/2722039034 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f4954003d60 con 0x7f496409c8c0 2024-03-29T11:12:42.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.975+0000 7f494bfff640 1 -- 172.21.15.121:0/2722039034 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 1234884141 0 0) 0x7f4954004600 con 0x7f496409c8c0 2024-03-29T11:12:42.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.975+0000 7f494bfff640 10 client.4152.objecter ms_dispatch 0x7f4964000b90 osd_map(8..8 src has 1..8) v4 2024-03-29T11:12:42.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.975+0000 7f494bfff640 3 client.4152.objecter handle_osd_map got epochs [8,8] > 0 2024-03-29T11:12:42.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.975+0000 7f494bfff640 3 client.4152.objecter handle_osd_map decoding full epoch 8 2024-03-29T11:12:42.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.975+0000 7f494bfff640 20 client.4152.objecter dump_active .. 0 homeless 2024-03-29T11:12:42.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.976+0000 7f4969a0d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:42.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.976+0000 7f4969a0d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:42.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.976+0000 7f4969a0d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:12:42.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.976+0000 7f4969a0d640 1 -- 172.21.15.121:0/2722039034 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f496409b170 con 0x7f496409c8c0 2024-03-29T11:12:42.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.978+0000 7f494bfff640 1 -- 172.21.15.121:0/2722039034 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f49540049a0 con 0x7f496409c8c0 2024-03-29T11:12:42.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.978+0000 7f494bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:42.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:42.978+0000 7f494bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:12:43.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.050+0000 7f4969a0d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "c06ac13d-25ef-476f-8acf-0f90ad4b2a00", "id": 6}] 2024-03-29T11:12:43.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.050+0000 7f4969a0d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "c06ac13d-25ef-476f-8acf-0f90ad4b2a00", "id": 6}] 2024-03-29T11:12:43.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.050+0000 7f4969a0d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:12:43.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.050+0000 7f4969a0d640 1 -- 172.21.15.121:0/2722039034 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd new", "uuid": "c06ac13d-25ef-476f-8acf-0f90ad4b2a00", "id": 6} v 0) v1 -- 0x7f496404f730 con 0x7f496409c8c0 2024-03-29T11:12:43.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.051+0000 7f494bfff640 1 -- 172.21.15.121:0/2722039034 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c06ac13d-25ef-476f-8acf-0f90ad4b2a00", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 1984269928 0 1113811849) 0x7f49540040e0 con 0x7f496409c8c0 2024-03-29T11:12:43.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.051+0000 7f494bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "c06ac13d-25ef-476f-8acf-0f90ad4b2a00", "id": 6}] 2024-03-29T11:12:43.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.051+0000 7f494bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:12:43.029 INFO:teuthology.orchestra.run.smithi121.stdout:6 2024-03-29T11:12:43.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.052+0000 7f4969a0d640 10 librados: watch_flush enter 2024-03-29T11:12:43.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.052+0000 7f4969a0d640 10 librados: watch_flush exit 2024-03-29T11:12:43.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.052+0000 7f4969a0d640 20 client.4152.objecter shutdown clearing up homeless session... 2024-03-29T11:12:43.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.052+0000 7f4969a0d640 10 client.4152.objecter successfully canceled tick 2024-03-29T11:12:43.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.052+0000 7f4969a0d640 10 monclient: shutdown 2024-03-29T11:12:43.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.052+0000 7f4969a0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:43.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.052+0000 7f4969a0d640 1 -- 172.21.15.121:0/2722039034 >> v1:172.21.15.47:6789/0 conn(0x7f496409c8c0 legacy=0x7f4964131140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:43.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.052+0000 7f4969a0d640 1 -- 172.21.15.121:0/2722039034 shutdown_connections 2024-03-29T11:12:43.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.052+0000 7f4969a0d640 1 -- 172.21.15.121:0/2722039034 >> 172.21.15.121:0/2722039034 conn(0x7f4964066b00 msgr2=0x7f496406b230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:43.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.052+0000 7f4969a0d640 1 -- 172.21.15.121:0/2722039034 shutdown_connections 2024-03-29T11:12:43.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.052+0000 7f4969a0d640 1 -- 172.21.15.121:0/2722039034 wait complete. 2024-03-29T11:12:43.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.052+0000 7f4969a0d640 1 librados: shutdown 2024-03-29T11:12:43.040 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new 52478c03-0b80-454b-ba89-4a8228777210 7 2024-03-29T11:12:43.044 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-29T11:12:43.066+0000 7fcab480f040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:12:43.097 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-29T11:12:43.119+0000 7f3465a3c040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:12:43.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.163+0000 7fa31af3d640 10 monclient: get_monmap_and_config 2024-03-29T11:12:43.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.163+0000 7fa31af3d640 10 monclient: build_initial_monmap 2024-03-29T11:12:43.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.163+0000 7fa31af3d640 10 monclient: monmap: 2024-03-29T11:12:43.141 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-03-29T11:12:43.141 INFO:teuthology.orchestra.run.smithi121.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:43.141 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:12:43.164081+0000 2024-03-29T11:12:43.141 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:12:43.164081+0000 2024-03-29T11:12:43.141 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:43.142 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:43.142 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:12:43.142 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:12:43.142 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:12:43.142 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:43.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.163+0000 7fa31af3d640 1 Processor -- start 2024-03-29T11:12:43.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.163+0000 7fa31af3d640 1 -- start start 2024-03-29T11:12:43.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.163+0000 7fa31af3d640 10 monclient: init 2024-03-29T11:12:43.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.164+0000 7fa31af3d640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:43.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.164+0000 7fa31af3d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:12:43.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.164+0000 7fa31af3d640 10 monclient(hunting): picked mon.noname-c con 0x7fa31409ac40 addr v1:172.21.15.121:6789/0 2024-03-29T11:12:43.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.164+0000 7fa31af3d640 10 monclient(hunting): picked mon.noname-b con 0x7fa31409c870 addr v1:172.21.15.47:6790/0 2024-03-29T11:12:43.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.164+0000 7fa31af3d640 10 monclient(hunting): picked mon.noname-a con 0x7fa3140b4360 addr v1:172.21.15.47:6789/0 2024-03-29T11:12:43.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.164+0000 7fa31af3d640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa31409dec0 con 0x7fa3140b4360 2024-03-29T11:12:43.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.164+0000 7fa31af3d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa31409da30 con 0x7fa31409ac40 2024-03-29T11:12:43.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.164+0000 7fa31af3d640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3140b39c0 con 0x7fa31409c870 2024-03-29T11:12:43.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.164+0000 7fa31af3d640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:43.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.164+0000 7fa31af3d640 10 monclient(hunting): authenticate will time out at 817.330685s 2024-03-29T11:12:43.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.164+0000 7fa318cd9640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fa31409ac40 0x7fa3140b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:54734/0 (socket says 172.21.15.121:54734) 2024-03-29T11:12:43.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.164+0000 7fa318cd9640 1 -- 172.21.15.121:0/2434144873 learned_addr learned my addr 172.21.15.121:0/2434144873 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-03-29T11:12:43.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.164+0000 7fa3137fe640 1 -- 172.21.15.121:0/2434144873 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 274087624 0 0) 0x7fa31409da30 con 0x7fa31409ac40 2024-03-29T11:12:43.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.164+0000 7fa3137fe640 10 monclient(hunting): _negotiate 2024-03-29T11:12:43.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.164+0000 7fa3137fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:43.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.164+0000 7fa3137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:43.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.164+0000 7fa3137fe640 10 monclient(hunting): my global_id is 4150 2024-03-29T11:12:43.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.164+0000 7fa3137fe640 1 -- 172.21.15.121:0/2434144873 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa304003520 con 0x7fa31409ac40 2024-03-29T11:12:43.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.165+0000 7fa3137fe640 1 -- 172.21.15.121:0/2434144873 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3212409888 0 0) 0x7fa3140b39c0 con 0x7fa31409c870 2024-03-29T11:12:43.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.165+0000 7fa3137fe640 10 monclient(hunting): _negotiate 2024-03-29T11:12:43.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.165+0000 7fa3137fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:43.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.165+0000 7fa3137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:43.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.165+0000 7fa3137fe640 10 monclient(hunting): my global_id is 4154 2024-03-29T11:12:43.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.165+0000 7fa3137fe640 1 -- 172.21.15.121:0/2434144873 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa31409da30 con 0x7fa31409c870 2024-03-29T11:12:43.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.165+0000 7fa3137fe640 1 -- 172.21.15.121:0/2434144873 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 384001985 0 0) 0x7fa30c001490 con 0x7fa3140b4360 2024-03-29T11:12:43.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.165+0000 7fa3137fe640 10 monclient(hunting): _negotiate 2024-03-29T11:12:43.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.165+0000 7fa3137fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:43.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.165+0000 7fa3137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:43.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.165+0000 7fa3137fe640 10 monclient(hunting): my global_id is 4155 2024-03-29T11:12:43.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.165+0000 7fa3137fe640 1 -- 172.21.15.121:0/2434144873 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3140b39c0 con 0x7fa3140b4360 2024-03-29T11:12:43.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.165+0000 7fa3137fe640 1 -- 172.21.15.121:0/2434144873 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2855012325 0 0) 0x7fa304003520 con 0x7fa31409ac40 2024-03-29T11:12:43.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.165+0000 7fa3137fe640 1 -- 172.21.15.121:0/2434144873 >> v1:172.21.15.47:6790/0 conn(0x7fa31409c870 legacy=0x7fa31413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:43.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.165+0000 7fa3137fe640 1 -- 172.21.15.121:0/2434144873 >> v1:172.21.15.47:6789/0 conn(0x7fa3140b4360 legacy=0x7fa31414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:43.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.165+0000 7fa3137fe640 10 monclient: _finish_hunting 0 2024-03-29T11:12:43.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.165+0000 7fa3137fe640 1 monclient: found mon.noname-c 2024-03-29T11:12:43.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.165+0000 7fa3137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:43.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.165+0000 7fa3137fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:12:43.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.165+0000 7fa3137fe640 1 -- 172.21.15.121:0/2434144873 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3140b5160 con 0x7fa31409ac40 2024-03-29T11:12:43.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.165+0000 7fa3137fe640 10 monclient: _finish_auth 0 2024-03-29T11:12:43.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.165+0000 7fa3137fe640 10 monclient: _check_auth_tickets 2024-03-29T11:12:43.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.165+0000 7fa3137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:43.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.165+0000 7fa3137fe640 1 -- 172.21.15.121:0/2434144873 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fa308001690 con 0x7fa31409ac40 2024-03-29T11:12:43.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.165+0000 7fa31af3d640 5 monclient: authenticate success, global_id 4150 2024-03-29T11:12:43.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.165+0000 7fa31af3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:43.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.165+0000 7fa3137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:43.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.165+0000 7fa3137fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:12:43.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.165+0000 7fa3137fe640 10 monclient: dump: 2024-03-29T11:12:43.157 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:12:43.158 INFO:teuthology.orchestra.run.smithi121.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:43.158 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:43.158 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:43.158 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:43.158 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:43.158 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:43.158 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:43.158 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:43.158 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:43.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.165+0000 7fa31af3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:43.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.165+0000 7fa3137fe640 1 -- 172.21.15.121:0/2434144873 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa308001870 con 0x7fa31409ac40 2024-03-29T11:12:43.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.165+0000 7fa3137fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:43.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.165+0000 7fa3137fe640 1 -- 172.21.15.121:0/2434144873 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fa308001c80 con 0x7fa31409ac40 2024-03-29T11:12:43.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.165+0000 7fa31af3d640 10 monclient: get_monmap_and_config success 2024-03-29T11:12:43.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.165+0000 7fa31af3d640 4 set_mon_vals no callback set 2024-03-29T11:12:43.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.165+0000 7fa3137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:43.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.165+0000 7fa3137fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:43.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.165+0000 7fa3137fe640 10 monclient: dump: 2024-03-29T11:12:43.159 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:12:43.159 INFO:teuthology.orchestra.run.smithi121.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:43.159 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:43.159 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:43.159 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:43.159 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:43.159 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:43.159 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:43.159 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:43.159 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:43.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.165+0000 7fa31af3d640 10 monclient: shutdown 2024-03-29T11:12:43.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.165+0000 7fa31af3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:43.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.165+0000 7fa31af3d640 1 -- 172.21.15.121:0/2434144873 >> v1:172.21.15.121:6789/0 conn(0x7fa31409ac40 legacy=0x7fa3140b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:43.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.165+0000 7fa31af3d640 1 -- 172.21.15.121:0/2434144873 shutdown_connections 2024-03-29T11:12:43.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.165+0000 7fa31af3d640 1 -- 172.21.15.121:0/2434144873 >> 172.21.15.121:0/2434144873 conn(0x7fa314066b00 msgr2=0x7fa314066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:43.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.165+0000 7fa31af3d640 1 -- 172.21.15.121:0/2434144873 shutdown_connections 2024-03-29T11:12:43.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.165+0000 7fa31af3d640 1 -- 172.21.15.121:0/2434144873 wait complete. 2024-03-29T11:12:43.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.166+0000 7fa31af3d640 10 monclient: build_initial_monmap 2024-03-29T11:12:43.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.166+0000 7fa31af3d640 10 monclient: monmap: 2024-03-29T11:12:43.160 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-03-29T11:12:43.160 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:12:43.160 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-03-29T11:12:43.160 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-03-29T11:12:43.160 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:43.160 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:43.161 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:12:43.161 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:12:43.161 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:12:43.161 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:43.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.167+0000 7fa31af3d640 1 librados: starting msgr at 2024-03-29T11:12:43.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.167+0000 7fa31af3d640 1 librados: starting objecter 2024-03-29T11:12:43.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.167+0000 7fa31af3d640 1 Processor -- start 2024-03-29T11:12:43.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.167+0000 7fa31af3d640 1 -- start start 2024-03-29T11:12:43.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.167+0000 7fa31af3d640 1 librados: setting wanted keys 2024-03-29T11:12:43.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.167+0000 7fa31af3d640 1 librados: calling monclient init 2024-03-29T11:12:43.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.167+0000 7fa31af3d640 10 monclient: init 2024-03-29T11:12:43.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.168+0000 7fa31af3d640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:43.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.168+0000 7fa31af3d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:12:43.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.168+0000 7fa31af3d640 10 monclient(hunting): picked mon.noname-b con 0x7fa31409ac40 addr v1:172.21.15.121:6789/0 2024-03-29T11:12:43.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.168+0000 7fa31af3d640 10 monclient(hunting): picked mon.noname-c con 0x7fa3140af1d0 addr v1:172.21.15.47:6790/0 2024-03-29T11:12:43.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.168+0000 7fa31af3d640 10 monclient(hunting): picked mon.noname-a con 0x7fa314139370 addr v1:172.21.15.47:6789/0 2024-03-29T11:12:43.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.168+0000 7fa31af3d640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3141d4be0 con 0x7fa314139370 2024-03-29T11:12:43.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.168+0000 7fa31af3d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3140b06a0 con 0x7fa31409ac40 2024-03-29T11:12:43.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.168+0000 7fa31af3d640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3140b1910 con 0x7fa3140af1d0 2024-03-29T11:12:43.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.168+0000 7fa31af3d640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:43.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.168+0000 7fa31af3d640 10 monclient(hunting): authenticate will time out at 817.334693s 2024-03-29T11:12:43.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.168+0000 7fa318cd9640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fa31409ac40 0x7fa3140afdd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:54746/0 (socket says 172.21.15.121:54746) 2024-03-29T11:12:43.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.168+0000 7fa318cd9640 1 -- 172.21.15.121:0/2689942480 learned_addr learned my addr 172.21.15.121:0/2689942480 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-03-29T11:12:43.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.168+0000 7fa3117fa640 10 client.?.objecter ms_handle_connect 0x7fa31409ac40 2024-03-29T11:12:43.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.168+0000 7fa3117fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:43.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.168+0000 7fa3117fa640 10 client.?.objecter ms_handle_connect 0x7fa314139370 2024-03-29T11:12:43.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.168+0000 7fa3117fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:43.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.168+0000 7fa3117fa640 10 client.?.objecter ms_handle_connect 0x7fa3140af1d0 2024-03-29T11:12:43.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.168+0000 7fa3117fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:43.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.168+0000 7fa3117fa640 1 -- 172.21.15.121:0/2689942480 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1426446538 0 0) 0x7fa3140b06a0 con 0x7fa31409ac40 2024-03-29T11:12:43.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.168+0000 7fa3117fa640 10 monclient(hunting): _negotiate 2024-03-29T11:12:43.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.168+0000 7fa3117fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:43.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.168+0000 7fa3117fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:43.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.168+0000 7fa3117fa640 10 monclient(hunting): my global_id is 4153 2024-03-29T11:12:43.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.168+0000 7fa3117fa640 1 -- 172.21.15.121:0/2689942480 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2f8004eb0 con 0x7fa31409ac40 2024-03-29T11:12:43.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.168+0000 7fa3117fa640 1 -- 172.21.15.121:0/2689942480 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2123851788 0 0) 0x7fa3141d4be0 con 0x7fa314139370 2024-03-29T11:12:43.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.168+0000 7fa3117fa640 10 monclient(hunting): _negotiate 2024-03-29T11:12:43.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.168+0000 7fa3117fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:43.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.168+0000 7fa3117fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:43.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.168+0000 7fa3117fa640 10 monclient(hunting): my global_id is 4158 2024-03-29T11:12:43.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.168+0000 7fa3117fa640 1 -- 172.21.15.121:0/2689942480 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3140b06a0 con 0x7fa314139370 2024-03-29T11:12:43.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.168+0000 7fa3117fa640 1 -- 172.21.15.121:0/2689942480 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4261372602 0 0) 0x7fa3140b1910 con 0x7fa3140af1d0 2024-03-29T11:12:43.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.168+0000 7fa3117fa640 10 monclient(hunting): _negotiate 2024-03-29T11:12:43.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.168+0000 7fa3117fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:43.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.168+0000 7fa3117fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:43.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.168+0000 7fa3117fa640 10 monclient(hunting): my global_id is 4157 2024-03-29T11:12:43.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.168+0000 7fa3117fa640 1 -- 172.21.15.121:0/2689942480 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3141d4be0 con 0x7fa3140af1d0 2024-03-29T11:12:43.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.169+0000 7fa3117fa640 1 -- 172.21.15.121:0/2689942480 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1889611010 0 0) 0x7fa2f8004eb0 con 0x7fa31409ac40 2024-03-29T11:12:43.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.169+0000 7fa3117fa640 1 -- 172.21.15.121:0/2689942480 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa3140b1910 con 0x7fa31409ac40 2024-03-29T11:12:43.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.169+0000 7fa3117fa640 1 -- 172.21.15.121:0/2689942480 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fa308001bc0 con 0x7fa31409ac40 2024-03-29T11:12:43.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.169+0000 7fa3117fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:43.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.169+0000 7fa3117fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:12:43.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.169+0000 7fa3117fa640 10 monclient(hunting): dump: 2024-03-29T11:12:43.164 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:12:43.164 INFO:teuthology.orchestra.run.smithi121.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:43.164 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:43.164 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:43.164 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:43.164 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:43.165 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:43.165 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:43.165 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:43.165 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:43.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.169+0000 7fa3117fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:12:43.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.169+0000 7fa3117fa640 1 -- 172.21.15.121:0/2689942480 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2616930766 0 0) 0x7fa3141d4be0 con 0x7fa3140af1d0 2024-03-29T11:12:43.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.169+0000 7fa3117fa640 1 -- 172.21.15.121:0/2689942480 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2f8004eb0 con 0x7fa3140af1d0 2024-03-29T11:12:43.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.169+0000 7fa3117fa640 1 -- 172.21.15.121:0/2689942480 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1620879129 0 0) 0x7fa3140b06a0 con 0x7fa314139370 2024-03-29T11:12:43.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.169+0000 7fa3117fa640 1 -- 172.21.15.121:0/2689942480 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa3141d4be0 con 0x7fa314139370 2024-03-29T11:12:43.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.169+0000 7fa3117fa640 1 -- 172.21.15.121:0/2689942480 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fa300001870 con 0x7fa3140af1d0 2024-03-29T11:12:43.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.169+0000 7fa3117fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:43.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.169+0000 7fa3117fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:12:43.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.169+0000 7fa3117fa640 10 monclient(hunting): dump: 2024-03-29T11:12:43.165 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:12:43.165 INFO:teuthology.orchestra.run.smithi121.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:43.166 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:43.166 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:43.166 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:43.166 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:43.166 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:43.166 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:43.166 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:43.166 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:43.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.169+0000 7fa3117fa640 1 -- 172.21.15.121:0/2689942480 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fa30c001b40 con 0x7fa314139370 2024-03-29T11:12:43.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.169+0000 7fa3117fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:43.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.169+0000 7fa3117fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:12:43.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.169+0000 7fa3117fa640 10 monclient(hunting): dump: 2024-03-29T11:12:43.166 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:12:43.166 INFO:teuthology.orchestra.run.smithi121.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:43.166 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:43.167 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:43.167 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:43.167 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:43.167 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:43.167 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:43.167 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:43.167 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:43.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.169+0000 7fa3117fa640 1 -- 172.21.15.121:0/2689942480 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4062557336 0 0) 0x7fa3140b1910 con 0x7fa31409ac40 2024-03-29T11:12:43.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.169+0000 7fa3117fa640 1 -- 172.21.15.121:0/2689942480 >> v1:172.21.15.47:6790/0 conn(0x7fa3140af1d0 legacy=0x7fa314138c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:43.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.169+0000 7fa3117fa640 1 -- 172.21.15.121:0/2689942480 >> v1:172.21.15.47:6789/0 conn(0x7fa314139370 legacy=0x7fa3141d70e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:43.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.169+0000 7fa3117fa640 10 monclient: _finish_hunting 0 2024-03-29T11:12:43.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.169+0000 7fa3117fa640 1 monclient: found mon.b 2024-03-29T11:12:43.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.169+0000 7fa3117fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:43.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.169+0000 7fa3117fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:12:43.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.169+0000 7fa3117fa640 1 -- 172.21.15.121:0/2689942480 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa314139970 con 0x7fa31409ac40 2024-03-29T11:12:43.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.169+0000 7fa3117fa640 10 monclient: _finish_auth 0 2024-03-29T11:12:43.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.169+0000 7fa3117fa640 10 monclient: _check_auth_tickets 2024-03-29T11:12:43.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.169+0000 7fa3117fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:43.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.169+0000 7fa31af3d640 5 monclient: authenticate success, global_id 4153 2024-03-29T11:12:43.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.169+0000 7fa31af3d640 10 monclient: _renew_subs 2024-03-29T11:12:43.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.169+0000 7fa31af3d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:12:43.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.169+0000 7fa31af3d640 1 -- 172.21.15.121:0/2689942480 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3141d77f0 con 0x7fa31409ac40 2024-03-29T11:12:43.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.169+0000 7fa31af3d640 10 client.4153.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:12:43.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.169+0000 7fa31af3d640 10 monclient: _renew_subs 2024-03-29T11:12:43.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.169+0000 7fa31af3d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:12:43.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.169+0000 7fa31af3d640 1 -- 172.21.15.121:0/2689942480 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3141d7d20 con 0x7fa31409ac40 2024-03-29T11:12:43.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.169+0000 7fa31af3d640 1 librados: init done 2024-03-29T11:12:43.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.169+0000 7fa3117fa640 1 -- 172.21.15.121:0/2689942480 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa308004bf0 con 0x7fa31409ac40 2024-03-29T11:12:43.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.169+0000 7fa3117fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:43.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.169+0000 7fa3117fa640 1 -- 172.21.15.121:0/2689942480 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fa308004f00 con 0x7fa31409ac40 2024-03-29T11:12:43.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.169+0000 7fa3117fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:43.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.169+0000 7fa3117fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:43.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.169+0000 7fa312ffd640 4 set_mon_vals no callback set 2024-03-29T11:12:43.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.169+0000 7fa3117fa640 10 monclient: dump: 2024-03-29T11:12:43.169 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-03-29T11:12:43.169 INFO:teuthology.orchestra.run.smithi121.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:43.169 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:43.170 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:43.170 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:43.170 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-03-29T11:12:43.170 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:43.170 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:43.170 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:43.170 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:12:43.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.169+0000 7fa3117fa640 1 -- 172.21.15.121:0/2689942480 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fa3080053f0 con 0x7fa31409ac40 2024-03-29T11:12:43.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.169+0000 7fa3117fa640 1 -- 172.21.15.121:0/2689942480 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 4077944849 0 0) 0x7fa308006a20 con 0x7fa31409ac40 2024-03-29T11:12:43.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.169+0000 7fa3117fa640 10 client.4153.objecter ms_dispatch 0x7fa314000b90 osd_map(9..9 src has 1..9) v4 2024-03-29T11:12:43.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.169+0000 7fa3117fa640 3 client.4153.objecter handle_osd_map got epochs [9,9] > 0 2024-03-29T11:12:43.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.169+0000 7fa3117fa640 3 client.4153.objecter handle_osd_map decoding full epoch 9 2024-03-29T11:12:43.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.169+0000 7fa3117fa640 20 client.4153.objecter dump_active .. 0 homeless 2024-03-29T11:12:43.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.170+0000 7fa31af3d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:43.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.170+0000 7fa31af3d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:43.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.170+0000 7fa31af3d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:12:43.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.170+0000 7fa31af3d640 1 -- 172.21.15.121:0/2689942480 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2f4005080 con 0x7fa31409ac40 2024-03-29T11:12:43.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.172+0000 7fa3117fa640 1 -- 172.21.15.121:0/2689942480 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fa308006be0 con 0x7fa31409ac40 2024-03-29T11:12:43.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.172+0000 7fa3117fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:43.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.172+0000 7fa3117fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:12:43.184 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-29T11:12:43.206+0000 7fcab480f040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:12:43.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.245+0000 7fa31af3d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "52478c03-0b80-454b-ba89-4a8228777210", "id": 7}] 2024-03-29T11:12:43.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.245+0000 7fa31af3d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "52478c03-0b80-454b-ba89-4a8228777210", "id": 7}] 2024-03-29T11:12:43.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.245+0000 7fa31af3d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:12:43.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.245+0000 7fa31af3d640 1 -- 172.21.15.121:0/2689942480 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd new", "uuid": "52478c03-0b80-454b-ba89-4a8228777210", "id": 7} v 0) v1 -- 0x7fa2f40058b0 con 0x7fa31409ac40 2024-03-29T11:12:43.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.248+0000 7fa3117fa640 1 -- 172.21.15.121:0/2689942480 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "52478c03-0b80-454b-ba89-4a8228777210", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 3246273388 0 1371665406) 0x7fa308006540 con 0x7fa31409ac40 2024-03-29T11:12:43.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.248+0000 7fa3117fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "52478c03-0b80-454b-ba89-4a8228777210", "id": 7}] 2024-03-29T11:12:43.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.248+0000 7fa3117fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:12:43.226 INFO:teuthology.orchestra.run.smithi121.stdout:7 2024-03-29T11:12:43.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.249+0000 7fa31af3d640 10 librados: watch_flush enter 2024-03-29T11:12:43.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.249+0000 7fa31af3d640 10 librados: watch_flush exit 2024-03-29T11:12:43.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.249+0000 7fa31af3d640 20 client.4153.objecter shutdown clearing up homeless session... 2024-03-29T11:12:43.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.249+0000 7fa31af3d640 10 client.4153.objecter successfully canceled tick 2024-03-29T11:12:43.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.249+0000 7fa31af3d640 10 monclient: shutdown 2024-03-29T11:12:43.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.249+0000 7fa31af3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:43.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.249+0000 7fa31af3d640 1 -- 172.21.15.121:0/2689942480 >> v1:172.21.15.121:6789/0 conn(0x7fa31409ac40 legacy=0x7fa3140afdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:43.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.249+0000 7fa31af3d640 1 -- 172.21.15.121:0/2689942480 shutdown_connections 2024-03-29T11:12:43.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.249+0000 7fa31af3d640 1 -- 172.21.15.121:0/2689942480 >> 172.21.15.121:0/2689942480 conn(0x7fa314066b00 msgr2=0x7fa3140b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:43.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.249+0000 7fa31af3d640 1 -- 172.21.15.121:0/2689942480 shutdown_connections 2024-03-29T11:12:43.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.249+0000 7fa31af3d640 1 -- 172.21.15.121:0/2689942480 wait complete. 2024-03-29T11:12:43.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-29T11:12:43.249+0000 7fa31af3d640 1 librados: shutdown 2024-03-29T11:12:43.237 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-29T11:12:43.237 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-29T11:12:43.239 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-29T11:12:43.261+0000 7f3465a3c040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:12:43.240 INFO:tasks.ceph.osd.0:Started 2024-03-29T11:12:43.240 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-29T11:12:43.240 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-29T11:12:43.242 INFO:tasks.ceph.osd.1:Started 2024-03-29T11:12:43.242 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-29T11:12:43.242 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-29T11:12:43.244 INFO:tasks.ceph.osd.2:Started 2024-03-29T11:12:43.244 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-29T11:12:43.245 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-29T11:12:43.247 INFO:tasks.ceph.osd.3:Started 2024-03-29T11:12:43.247 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-29T11:12:43.247 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-29T11:12:43.249 INFO:tasks.ceph.osd.4:Started 2024-03-29T11:12:43.249 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-29T11:12:43.249 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-29T11:12:43.251 INFO:tasks.ceph.osd.5:Started 2024-03-29T11:12:43.251 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-29T11:12:43.251 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-29T11:12:43.253 INFO:tasks.ceph.osd.6:Started 2024-03-29T11:12:43.253 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-29T11:12:43.253 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-29T11:12:43.255 INFO:tasks.ceph.osd.7:Started 2024-03-29T11:12:43.256 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:12:43.345 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-29T11:12:43.367+0000 7fcab480f040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:12:43.376 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-29T11:12:43.398+0000 7f3465a3c040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:12:43.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.403+0000 7f919b97b640 10 monclient: get_monmap_and_config 2024-03-29T11:12:43.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.403+0000 7f919b97b640 10 monclient: build_initial_monmap 2024-03-29T11:12:43.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.403+0000 7f919b97b640 10 monclient: monmap: 2024-03-29T11:12:43.381 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-29T11:12:43.381 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:43.381 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-29T11:12:43.404604+0000 2024-03-29T11:12:43.381 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-29T11:12:43.404604+0000 2024-03-29T11:12:43.381 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:43.382 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-29T11:12:43.382 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:12:43.382 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:12:43.382 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:12:43.382 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-29T11:12:43.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.404+0000 7f919b97b640 1 Processor -- start 2024-03-29T11:12:43.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.404+0000 7f919b97b640 1 -- start start 2024-03-29T11:12:43.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.404+0000 7f919b97b640 10 monclient: init 2024-03-29T11:12:43.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.404+0000 7f919b97b640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:43.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.404+0000 7f919b97b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:12:43.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.405+0000 7f919b97b640 10 monclient(hunting): picked mon.noname-a con 0x7f91940b8240 addr v1:172.21.15.47:6789/0 2024-03-29T11:12:43.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.405+0000 7f919b97b640 10 monclient(hunting): picked mon.noname-b con 0x7f91940b1f60 addr v1:172.21.15.47:6790/0 2024-03-29T11:12:43.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.405+0000 7f919b97b640 10 monclient(hunting): picked mon.noname-c con 0x7f919409c8c0 addr v1:172.21.15.121:6789/0 2024-03-29T11:12:43.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.405+0000 7f919b97b640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f919409bd80 con 0x7f91940b8240 2024-03-29T11:12:43.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.405+0000 7f919b97b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f919409b960 con 0x7f919409c8c0 2024-03-29T11:12:43.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.405+0000 7f919b97b640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91940b2ec0 con 0x7f91940b1f60 2024-03-29T11:12:43.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.405+0000 7f919b97b640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:43.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.405+0000 7f919b97b640 10 monclient(hunting): authenticate will time out at 806.593088s 2024-03-29T11:12:43.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.405+0000 7f9199717640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f91940b8240 0x7f919413f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:56324/0 (socket says 172.21.15.47:56324) 2024-03-29T11:12:43.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.405+0000 7f9199717640 1 -- 172.21.15.47:0/4160647586 learned_addr learned my addr 172.21.15.47:0/4160647586 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:12:43.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.405+0000 7f918bfff640 1 -- 172.21.15.47:0/4160647586 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3399187230 0 0) 0x7f919409bd80 con 0x7f91940b8240 2024-03-29T11:12:43.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.405+0000 7f918bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:12:43.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.405+0000 7f918bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:43.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.405+0000 7f918bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:43.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.405+0000 7f918bfff640 10 monclient(hunting): my global_id is 4185 2024-03-29T11:12:43.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.405+0000 7f918bfff640 1 -- 172.21.15.47:0/4160647586 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9180003520 con 0x7f91940b8240 2024-03-29T11:12:43.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.405+0000 7f918bfff640 1 -- 172.21.15.47:0/4160647586 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3529030265 0 0) 0x7f91940b2ec0 con 0x7f91940b1f60 2024-03-29T11:12:43.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.405+0000 7f918bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:12:43.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.405+0000 7f918bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:43.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.405+0000 7f918bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:43.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.405+0000 7f918bfff640 10 monclient(hunting): my global_id is 4184 2024-03-29T11:12:43.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.405+0000 7f918bfff640 1 -- 172.21.15.47:0/4160647586 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f919409bd80 con 0x7f91940b1f60 2024-03-29T11:12:43.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.405+0000 7f918bfff640 1 -- 172.21.15.47:0/4160647586 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2494477016 0 0) 0x7f9180003520 con 0x7f91940b8240 2024-03-29T11:12:43.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.405+0000 7f918bfff640 1 -- 172.21.15.47:0/4160647586 >> v1:172.21.15.47:6790/0 conn(0x7f91940b1f60 legacy=0x7f919409c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:43.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.405+0000 7f918bfff640 1 -- 172.21.15.47:0/4160647586 >> v1:172.21.15.121:6789/0 conn(0x7f919409c8c0 legacy=0x7f919414d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:43.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.405+0000 7f918bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:12:43.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.405+0000 7f918bfff640 1 monclient: found mon.noname-a 2024-03-29T11:12:43.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.405+0000 7f918bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:43.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.405+0000 7f918bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-29T11:12:43.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.405+0000 7f918bfff640 1 -- 172.21.15.47:0/4160647586 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f919413fb60 con 0x7f91940b8240 2024-03-29T11:12:43.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.405+0000 7f918bfff640 10 monclient: _finish_auth 0 2024-03-29T11:12:43.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.405+0000 7f918bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:12:43.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.405+0000 7f918bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:43.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.405+0000 7f918bfff640 1 -- 172.21.15.47:0/4160647586 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f9190001690 con 0x7f91940b8240 2024-03-29T11:12:43.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.405+0000 7f918bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:43.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.405+0000 7f918bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:12:43.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.405+0000 7f918bfff640 10 monclient: dump: 2024-03-29T11:12:43.394 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-29T11:12:43.394 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:43.394 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:43.394 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:43.394 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:43.394 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-29T11:12:43.394 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:43.395 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:43.395 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:43.395 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-29T11:12:43.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.405+0000 7f919b97b640 5 monclient: authenticate success, global_id 4185 2024-03-29T11:12:43.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.405+0000 7f919b97b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:43.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.405+0000 7f918bfff640 1 -- 172.21.15.47:0/4160647586 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f91900017f0 con 0x7f91940b8240 2024-03-29T11:12:43.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.405+0000 7f918bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:43.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.405+0000 7f918bfff640 1 -- 172.21.15.47:0/4160647586 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f9190001c00 con 0x7f91940b8240 2024-03-29T11:12:43.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.405+0000 7f918bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:43.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.405+0000 7f918bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:12:43.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.405+0000 7f918bfff640 10 monclient: dump: 2024-03-29T11:12:43.395 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-29T11:12:43.395 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:43.395 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:43.395 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:43.396 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:43.396 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-29T11:12:43.396 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:43.396 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:43.396 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:43.396 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-29T11:12:43.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.405+0000 7f919b97b640 10 monclient: get_monmap_and_config success 2024-03-29T11:12:43.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.405+0000 7f919b97b640 4 set_mon_vals no callback set 2024-03-29T11:12:43.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.406+0000 7f919b97b640 10 monclient: shutdown 2024-03-29T11:12:43.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.406+0000 7f919b97b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:43.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.406+0000 7f919b97b640 1 -- 172.21.15.47:0/4160647586 >> v1:172.21.15.47:6789/0 conn(0x7f91940b8240 legacy=0x7f919413f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:43.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.406+0000 7f919b97b640 1 -- 172.21.15.47:0/4160647586 shutdown_connections 2024-03-29T11:12:43.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.406+0000 7f919b97b640 1 -- 172.21.15.47:0/4160647586 >> 172.21.15.47:0/4160647586 conn(0x7f9194066b00 msgr2=0x7f9194066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:43.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.406+0000 7f919b97b640 1 -- 172.21.15.47:0/4160647586 shutdown_connections 2024-03-29T11:12:43.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.406+0000 7f919b97b640 1 -- 172.21.15.47:0/4160647586 wait complete. 2024-03-29T11:12:43.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.407+0000 7f919b97b640 10 monclient: build_initial_monmap 2024-03-29T11:12:43.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.407+0000 7f919b97b640 10 monclient: monmap: 2024-03-29T11:12:43.397 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-29T11:12:43.397 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:12:43.397 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-29T11:12:43.397 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-29T11:12:43.397 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:43.397 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-29T11:12:43.397 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:12:43.397 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:12:43.397 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:12:43.397 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-29T11:12:43.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.407+0000 7f919b97b640 1 librados: starting msgr at 2024-03-29T11:12:43.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.407+0000 7f919b97b640 1 librados: starting objecter 2024-03-29T11:12:43.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.407+0000 7f919b97b640 1 Processor -- start 2024-03-29T11:12:43.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.407+0000 7f919b97b640 1 -- start start 2024-03-29T11:12:43.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.408+0000 7f919b97b640 1 librados: setting wanted keys 2024-03-29T11:12:43.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.408+0000 7f919b97b640 1 librados: calling monclient init 2024-03-29T11:12:43.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.408+0000 7f919b97b640 10 monclient: init 2024-03-29T11:12:43.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.408+0000 7f919b97b640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:43.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.408+0000 7f919b97b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:12:43.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.408+0000 7f919b97b640 10 monclient(hunting): picked mon.noname-b con 0x7f919409c8c0 addr v1:172.21.15.121:6789/0 2024-03-29T11:12:43.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.408+0000 7f919b97b640 10 monclient(hunting): picked mon.noname-a con 0x7f91940b8240 addr v1:172.21.15.47:6789/0 2024-03-29T11:12:43.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.408+0000 7f919b97b640 10 monclient(hunting): picked mon.noname-c con 0x7f919413be70 addr v1:172.21.15.47:6790/0 2024-03-29T11:12:43.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.408+0000 7f919b97b640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f919413b770 con 0x7f91940b8240 2024-03-29T11:12:43.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.408+0000 7f919b97b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f919413c350 con 0x7f919409c8c0 2024-03-29T11:12:43.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.408+0000 7f9199f18640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f919413be70 0x7f91941e1980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:54876/0 (socket says 172.21.15.47:54876) 2024-03-29T11:12:43.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.408+0000 7f9199f18640 1 -- 172.21.15.47:0/1486602766 learned_addr learned my addr 172.21.15.47:0/1486602766 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:12:43.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.408+0000 7f919b97b640 1 -- 172.21.15.47:0/1486602766 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f919413c870 con 0x7f919413be70 2024-03-29T11:12:43.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.408+0000 7f919b97b640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:43.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.408+0000 7f9189ffb640 10 client.?.objecter ms_handle_connect 0x7f919413be70 2024-03-29T11:12:43.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.408+0000 7f9189ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:43.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.408+0000 7f9189ffb640 10 client.?.objecter ms_handle_connect 0x7f91940b8240 2024-03-29T11:12:43.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.408+0000 7f9189ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:43.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.408+0000 7f9189ffb640 10 client.?.objecter ms_handle_connect 0x7f919409c8c0 2024-03-29T11:12:43.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.408+0000 7f9189ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:43.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.408+0000 7f919b97b640 10 monclient(hunting): authenticate will time out at 806.596883s 2024-03-29T11:12:43.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f9189ffb640 1 -- 172.21.15.47:0/1486602766 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 711007827 0 0) 0x7f919413c350 con 0x7f919409c8c0 2024-03-29T11:12:43.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f9189ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:12:43.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f9189ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:43.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f9189ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:43.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f9189ffb640 10 monclient(hunting): my global_id is 4183 2024-03-29T11:12:43.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f9189ffb640 1 -- 172.21.15.47:0/1486602766 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9178004eb0 con 0x7f919409c8c0 2024-03-29T11:12:43.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f9189ffb640 1 -- 172.21.15.47:0/1486602766 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1955353656 0 0) 0x7f919413b770 con 0x7f91940b8240 2024-03-29T11:12:43.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f9189ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:12:43.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f9189ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:43.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f9189ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:43.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f9189ffb640 10 monclient(hunting): my global_id is 4188 2024-03-29T11:12:43.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f9189ffb640 1 -- 172.21.15.47:0/1486602766 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f919413c350 con 0x7f91940b8240 2024-03-29T11:12:43.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f9189ffb640 1 -- 172.21.15.47:0/1486602766 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 267291420 0 0) 0x7f919413c870 con 0x7f919413be70 2024-03-29T11:12:43.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f9189ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:12:43.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f9189ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:43.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f9189ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:43.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f9189ffb640 10 monclient(hunting): my global_id is 4187 2024-03-29T11:12:43.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f9189ffb640 1 -- 172.21.15.47:0/1486602766 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f919413b770 con 0x7f919413be70 2024-03-29T11:12:43.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f9189ffb640 1 -- 172.21.15.47:0/1486602766 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 623076110 0 0) 0x7f9178004eb0 con 0x7f919409c8c0 2024-03-29T11:12:43.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f9189ffb640 1 -- 172.21.15.47:0/1486602766 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f919413c870 con 0x7f919409c8c0 2024-03-29T11:12:43.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f9189ffb640 1 -- 172.21.15.47:0/1486602766 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f9190001b00 con 0x7f919409c8c0 2024-03-29T11:12:43.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f9189ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:43.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f9189ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:12:43.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f9189ffb640 10 monclient(hunting): dump: 2024-03-29T11:12:43.401 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-29T11:12:43.401 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:43.401 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:43.401 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:43.401 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:43.401 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-29T11:12:43.401 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:43.401 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:43.401 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:43.401 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-29T11:12:43.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f9189ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:12:43.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f9189ffb640 1 -- 172.21.15.47:0/1486602766 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3561355525 0 0) 0x7f919413c350 con 0x7f91940b8240 2024-03-29T11:12:43.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f9189ffb640 1 -- 172.21.15.47:0/1486602766 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9178004eb0 con 0x7f91940b8240 2024-03-29T11:12:43.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f9189ffb640 1 -- 172.21.15.47:0/1486602766 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f9184001830 con 0x7f91940b8240 2024-03-29T11:12:43.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f9189ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:43.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f9189ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:12:43.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f9189ffb640 10 monclient(hunting): dump: 2024-03-29T11:12:43.402 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-29T11:12:43.402 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:43.402 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:43.402 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:43.402 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:43.402 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-29T11:12:43.402 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:43.402 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:43.402 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:43.403 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-29T11:12:43.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f9189ffb640 1 -- 172.21.15.47:0/1486602766 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2560468953 0 0) 0x7f919413b770 con 0x7f919413be70 2024-03-29T11:12:43.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f9189ffb640 1 -- 172.21.15.47:0/1486602766 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f919413c350 con 0x7f919413be70 2024-03-29T11:12:43.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f9189ffb640 1 -- 172.21.15.47:0/1486602766 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f918c001bb0 con 0x7f919413be70 2024-03-29T11:12:43.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f9189ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:43.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f9189ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:12:43.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f9189ffb640 10 monclient(hunting): dump: 2024-03-29T11:12:43.403 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-29T11:12:43.403 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:43.403 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:43.403 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:43.403 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:43.403 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-29T11:12:43.403 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:43.403 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:43.404 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:43.404 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-29T11:12:43.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f9189ffb640 1 -- 172.21.15.47:0/1486602766 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3953273860 0 0) 0x7f919413c870 con 0x7f919409c8c0 2024-03-29T11:12:43.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f9189ffb640 1 -- 172.21.15.47:0/1486602766 >> v1:172.21.15.47:6790/0 conn(0x7f919413be70 legacy=0x7f91941e1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:43.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f9189ffb640 1 -- 172.21.15.47:0/1486602766 >> v1:172.21.15.47:6789/0 conn(0x7f91940b8240 legacy=0x7f91941da440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:43.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f9189ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:12:43.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f9189ffb640 1 monclient: found mon.b 2024-03-29T11:12:43.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f9189ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:43.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f9189ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:12:43.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f9189ffb640 1 -- 172.21.15.47:0/1486602766 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91940b1670 con 0x7f919409c8c0 2024-03-29T11:12:43.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f9189ffb640 10 monclient: _finish_auth 0 2024-03-29T11:12:43.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f9189ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:12:43.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f9189ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:43.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f919b97b640 5 monclient: authenticate success, global_id 4183 2024-03-29T11:12:43.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f919b97b640 10 monclient: _renew_subs 2024-03-29T11:12:43.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f919b97b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:12:43.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f919b97b640 1 -- 172.21.15.47:0/1486602766 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91940b1920 con 0x7f919409c8c0 2024-03-29T11:12:43.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f9189ffb640 1 -- 172.21.15.47:0/1486602766 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9190004a90 con 0x7f919409c8c0 2024-03-29T11:12:43.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f9189ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:43.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f9189ffb640 1 -- 172.21.15.47:0/1486602766 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f9190004da0 con 0x7f919409c8c0 2024-03-29T11:12:43.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f9189ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:43.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f9189ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:43.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f9189ffb640 10 monclient: dump: 2024-03-29T11:12:43.406 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-29T11:12:43.406 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:43.406 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:43.406 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:43.406 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:43.406 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-29T11:12:43.406 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:43.407 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:43.407 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:43.407 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-29T11:12:43.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f918b7fe640 4 set_mon_vals no callback set 2024-03-29T11:12:43.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f919b97b640 10 client.4183.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:12:43.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f919b97b640 10 monclient: _renew_subs 2024-03-29T11:12:43.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f9189ffb640 1 -- 172.21.15.47:0/1486602766 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f91900052d0 con 0x7f919409c8c0 2024-03-29T11:12:43.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f919b97b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:12:43.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f919b97b640 1 -- 172.21.15.47:0/1486602766 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f91941dad30 con 0x7f919409c8c0 2024-03-29T11:12:43.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.409+0000 7f919b97b640 1 librados: init done 2024-03-29T11:12:43.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.410+0000 7f9189ffb640 1 -- 172.21.15.47:0/1486602766 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7f9190005c50 con 0x7f919409c8c0 2024-03-29T11:12:43.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.410+0000 7f9189ffb640 10 client.4183.objecter ms_dispatch 0x7f9194000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:12:43.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.410+0000 7f9189ffb640 3 client.4183.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:12:43.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.410+0000 7f9189ffb640 3 client.4183.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:12:43.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.410+0000 7f9189ffb640 20 client.4183.objecter dump_active .. 0 homeless 2024-03-29T11:12:43.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.410+0000 7f919b97b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:43.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.410+0000 7f919b97b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:43.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.410+0000 7f919b97b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:12:43.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.410+0000 7f919b97b640 1 -- 172.21.15.47:0/1486602766 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f919409b170 con 0x7f919409c8c0 2024-03-29T11:12:43.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.412+0000 7f9189ffb640 1 -- 172.21.15.47:0/1486602766 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f9190005f10 con 0x7f919409c8c0 2024-03-29T11:12:43.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.412+0000 7f9189ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:43.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.412+0000 7f9189ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:12:43.408 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-29T11:12:43.417+0000 7fcab480f040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:12:43.422 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-29T11:12:43.444+0000 7f3465a3c040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:12:43.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.474+0000 7f919b97b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:43.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.474+0000 7f919b97b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:43.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.474+0000 7f919b97b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:12:43.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.474+0000 7f919b97b640 1 -- 172.21.15.47:0/1486602766 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f919413c540 con 0x7f919409c8c0 2024-03-29T11:12:43.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.475+0000 7f9189ffb640 1 -- 172.21.15.47:0/1486602766 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7f9190005f10 con 0x7f919409c8c0 2024-03-29T11:12:43.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.475+0000 7f9189ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:43.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.475+0000 7f9189ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:12:43.454 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-29T11:12:43.454 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:12:43.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.476+0000 7f919b97b640 10 librados: watch_flush enter 2024-03-29T11:12:43.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.476+0000 7f919b97b640 10 librados: watch_flush exit 2024-03-29T11:12:43.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.476+0000 7f919b97b640 20 client.4183.objecter shutdown clearing up homeless session... 2024-03-29T11:12:43.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.476+0000 7f919b97b640 10 client.4183.objecter successfully canceled tick 2024-03-29T11:12:43.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.476+0000 7f919b97b640 10 monclient: shutdown 2024-03-29T11:12:43.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.476+0000 7f919b97b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:43.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.476+0000 7f919b97b640 1 -- 172.21.15.47:0/1486602766 >> v1:172.21.15.121:6789/0 conn(0x7f919409c8c0 legacy=0x7f91941452b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:43.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.476+0000 7f919b97b640 1 -- 172.21.15.47:0/1486602766 shutdown_connections 2024-03-29T11:12:43.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.476+0000 7f919b97b640 1 -- 172.21.15.47:0/1486602766 >> 172.21.15.47:0/1486602766 conn(0x7f9194066b00 msgr2=0x7f919413dac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:43.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.476+0000 7f919b97b640 1 -- 172.21.15.47:0/1486602766 shutdown_connections 2024-03-29T11:12:43.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.476+0000 7f919b97b640 1 -- 172.21.15.47:0/1486602766 wait complete. 2024-03-29T11:12:43.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-29T11:12:43.476+0000 7f919b97b640 1 librados: shutdown 2024-03-29T11:12:43.464 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-29T11:12:43.465 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-29T11:12:43.513 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-29T11:12:43.535+0000 7fcab480f040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:12:43.535 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-29T11:12:43.557+0000 7f3465a3c040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:12:43.612 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-29T11:12:43.634+0000 7fcab480f040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:12:43.631 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-29T11:12:43.653+0000 7f3465a3c040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:12:43.772 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-29T11:12:43.794+0000 7fcab480f040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:12:43.784 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-29T11:12:43.806+0000 7f3465a3c040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:12:43.815 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-29T11:12:43.838+0000 7fcab480f040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:12:43.826 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-29T11:12:43.848+0000 7f3465a3c040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:12:43.848 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-03-29T11:12:43.870+0000 7ffabb413640 -1 Falling back to public interface 2024-03-29T11:12:43.848 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-03-29T11:12:43.871+0000 7f9464211640 -1 Falling back to public interface 2024-03-29T11:12:43.854 INFO:tasks.ceph.osd.1.smithi047.stderr:2024-03-29T11:12:43.876+0000 7f45ce4d7640 -1 Falling back to public interface 2024-03-29T11:12:43.857 INFO:tasks.ceph.osd.3.smithi047.stderr:2024-03-29T11:12:43.879+0000 7ff0f4817640 -1 Falling back to public interface 2024-03-29T11:12:43.857 INFO:tasks.ceph.osd.2.smithi047.stderr:2024-03-29T11:12:43.879+0000 7f6f05c0d640 -1 Falling back to public interface 2024-03-29T11:12:43.857 INFO:tasks.ceph.osd.2.smithi047.stderr:2024-03-29T11:12:43.879+0000 7f6f0461d640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6805/0: (98) Address already in use 2024-03-29T11:12:43.862 INFO:tasks.ceph.osd.0.smithi047.stderr:2024-03-29T11:12:43.884+0000 7fb3c6d81640 -1 Falling back to public interface 2024-03-29T11:12:43.869 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-03-29T11:12:43.891+0000 7f4507622640 -1 Falling back to public interface 2024-03-29T11:12:43.873 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-03-29T11:12:43.895+0000 7f11c4011640 -1 Falling back to public interface 2024-03-29T11:12:43.979 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-29T11:12:44.001+0000 7fcab480f040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:12:43.985 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-29T11:12:44.007+0000 7f3465a3c040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:12:44.270 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:12:44.361 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.382+0000 7f9951a84640 10 monclient: get_monmap_and_config 2024-03-29T11:12:44.361 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.382+0000 7f9951a84640 10 monclient: build_initial_monmap 2024-03-29T11:12:44.361 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.382+0000 7f9951a84640 10 monclient: monmap: 2024-03-29T11:12:44.361 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:12:44.361 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:44.361 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:44.383892+0000 2024-03-29T11:12:44.361 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:44.383892+0000 2024-03-29T11:12:44.361 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:44.361 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:12:44.361 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:12:44.361 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:12:44.361 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:12:44.362 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:12:44.370 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.383+0000 7f9951a84640 1 Processor -- start 2024-03-29T11:12:44.370 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.383+0000 7f9951a84640 1 -- start start 2024-03-29T11:12:44.370 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.383+0000 7f9951a84640 10 monclient: init 2024-03-29T11:12:44.370 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.384+0000 7f9951a84640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:44.370 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.384+0000 7f9951a84640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:12:44.370 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.384+0000 7f9951a84640 10 monclient(hunting): picked mon.noname-a con 0x7f994c1457f0 addr v1:172.21.15.47:6789/0 2024-03-29T11:12:44.370 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.384+0000 7f9951a84640 10 monclient(hunting): picked mon.noname-c con 0x7f994c13f650 addr v1:172.21.15.121:6789/0 2024-03-29T11:12:44.370 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.384+0000 7f9951a84640 10 monclient(hunting): picked mon.noname-b con 0x7f994c0a9d70 addr v1:172.21.15.47:6790/0 2024-03-29T11:12:44.371 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.384+0000 7f9951a84640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f994c0aaef0 con 0x7f994c1457f0 2024-03-29T11:12:44.371 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.384+0000 7f9951a84640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f994c145bd0 con 0x7f994c13f650 2024-03-29T11:12:44.371 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.384+0000 7f9951a84640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f994c145eb0 con 0x7f994c0a9d70 2024-03-29T11:12:44.371 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.384+0000 7f9951a84640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:44.371 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.384+0000 7f9951a84640 10 monclient(hunting): authenticate will time out at 807.572321s 2024-03-29T11:12:44.371 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.384+0000 7f994b7fe640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f994c1457f0 0x7f994c13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:56334/0 (socket says 172.21.15.47:56334) 2024-03-29T11:12:44.371 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.384+0000 7f994b7fe640 1 -- 172.21.15.47:0/491257650 learned_addr learned my addr 172.21.15.47:0/491257650 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:12:44.371 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.384+0000 7f994a7fc640 1 -- 172.21.15.47:0/491257650 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 919256934 0 0) 0x7f994c0aaef0 con 0x7f994c1457f0 2024-03-29T11:12:44.371 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.384+0000 7f994a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:12:44.371 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.384+0000 7f994a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:44.371 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.384+0000 7f994a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:44.371 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.384+0000 7f994a7fc640 10 monclient(hunting): my global_id is 4191 2024-03-29T11:12:44.371 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.384+0000 7f994a7fc640 1 -- 172.21.15.47:0/491257650 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9938003520 con 0x7f994c1457f0 2024-03-29T11:12:44.371 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.384+0000 7f994a7fc640 1 -- 172.21.15.47:0/491257650 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 235002405 0 0) 0x7f994c145eb0 con 0x7f994c0a9d70 2024-03-29T11:12:44.372 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.384+0000 7f994a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:12:44.372 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.384+0000 7f994a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:44.372 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.384+0000 7f994a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:44.372 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.384+0000 7f994a7fc640 10 monclient(hunting): my global_id is 4190 2024-03-29T11:12:44.372 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.384+0000 7f994a7fc640 1 -- 172.21.15.47:0/491257650 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f994c0aaef0 con 0x7f994c0a9d70 2024-03-29T11:12:44.372 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.384+0000 7f994a7fc640 1 -- 172.21.15.47:0/491257650 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 902014798 0 0) 0x7f994c145bd0 con 0x7f994c13f650 2024-03-29T11:12:44.372 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.384+0000 7f994a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:12:44.372 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.384+0000 7f994a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:44.372 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.384+0000 7f994a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:44.372 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.384+0000 7f994a7fc640 10 monclient(hunting): my global_id is 4186 2024-03-29T11:12:44.372 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.384+0000 7f994a7fc640 1 -- 172.21.15.47:0/491257650 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f994c145eb0 con 0x7f994c13f650 2024-03-29T11:12:44.372 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.385+0000 7f994a7fc640 1 -- 172.21.15.47:0/491257650 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1174811324 0 0) 0x7f9938003520 con 0x7f994c1457f0 2024-03-29T11:12:44.372 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.385+0000 7f994a7fc640 1 -- 172.21.15.47:0/491257650 >> v1:172.21.15.47:6790/0 conn(0x7f994c0a9d70 legacy=0x7f994c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:44.372 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.385+0000 7f994a7fc640 1 -- 172.21.15.47:0/491257650 >> v1:172.21.15.121:6789/0 conn(0x7f994c13f650 legacy=0x7f994c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:44.372 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.385+0000 7f994a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:12:44.373 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.385+0000 7f994a7fc640 1 monclient: found mon.noname-a 2024-03-29T11:12:44.373 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.385+0000 7f994a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:44.373 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.385+0000 7f994a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-29T11:12:44.373 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.385+0000 7f994a7fc640 1 -- 172.21.15.47:0/491257650 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f994c146340 con 0x7f994c1457f0 2024-03-29T11:12:44.373 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.385+0000 7f994a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:12:44.373 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.385+0000 7f994a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:12:44.373 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.385+0000 7f994a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:44.373 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.385+0000 7f9951a84640 5 monclient: authenticate success, global_id 4191 2024-03-29T11:12:44.373 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.385+0000 7f9951a84640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:44.373 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.385+0000 7f994a7fc640 1 -- 172.21.15.47:0/491257650 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f9934001690 con 0x7f994c1457f0 2024-03-29T11:12:44.373 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.385+0000 7f994a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:44.373 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.385+0000 7f994a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:12:44.373 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.385+0000 7f994a7fc640 10 monclient: dump: 2024-03-29T11:12:44.373 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:12:44.373 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:44.374 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:44.374 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:44.374 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:44.374 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:12:44.374 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:44.374 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:44.374 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:44.374 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:12:44.374 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.385+0000 7f9951a84640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:44.374 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.385+0000 7f994a7fc640 1 -- 172.21.15.47:0/491257650 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9934001860 con 0x7f994c1457f0 2024-03-29T11:12:44.374 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.385+0000 7f994a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:44.374 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.385+0000 7f9951a84640 10 monclient: get_monmap_and_config success 2024-03-29T11:12:44.374 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.385+0000 7f994a7fc640 1 -- 172.21.15.47:0/491257650 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f9934001bf0 con 0x7f994c1457f0 2024-03-29T11:12:44.374 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.385+0000 7f994a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:44.375 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.385+0000 7f9951a84640 4 set_mon_vals no callback set 2024-03-29T11:12:44.375 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.385+0000 7f994a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:12:44.375 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.385+0000 7f994a7fc640 10 monclient: dump: 2024-03-29T11:12:44.375 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:12:44.375 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:44.375 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:44.375 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:44.375 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:44.375 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:12:44.375 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:44.375 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:44.375 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:44.375 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:12:44.375 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.385+0000 7f9951a84640 10 monclient: shutdown 2024-03-29T11:12:44.375 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.385+0000 7f9951a84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:44.375 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.385+0000 7f9951a84640 1 -- 172.21.15.47:0/491257650 >> v1:172.21.15.47:6789/0 conn(0x7f994c1457f0 legacy=0x7f994c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:44.376 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.385+0000 7f9951a84640 1 -- 172.21.15.47:0/491257650 shutdown_connections 2024-03-29T11:12:44.376 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.385+0000 7f9951a84640 1 -- 172.21.15.47:0/491257650 >> 172.21.15.47:0/491257650 conn(0x7f994c066a00 msgr2=0x7f994c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:44.376 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.385+0000 7f9951a84640 1 -- 172.21.15.47:0/491257650 shutdown_connections 2024-03-29T11:12:44.376 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.385+0000 7f9951a84640 1 -- 172.21.15.47:0/491257650 wait complete. 2024-03-29T11:12:44.376 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.386+0000 7f9951a84640 10 monclient: build_initial_monmap 2024-03-29T11:12:44.376 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.386+0000 7f9951a84640 10 monclient: monmap: 2024-03-29T11:12:44.376 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:12:44.376 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:12:44.376 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:12:44.376 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:12:44.376 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:44.376 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:12:44.376 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:12:44.376 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:12:44.377 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:12:44.377 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:12:44.377 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.386+0000 7f9951a84640 1 librados: starting msgr at 2024-03-29T11:12:44.377 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.386+0000 7f9951a84640 1 librados: starting objecter 2024-03-29T11:12:44.377 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.386+0000 7f9951a84640 1 Processor -- start 2024-03-29T11:12:44.377 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.387+0000 7f9951a84640 1 -- start start 2024-03-29T11:12:44.377 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.387+0000 7f9951a84640 1 librados: setting wanted keys 2024-03-29T11:12:44.377 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.387+0000 7f9951a84640 1 librados: calling monclient init 2024-03-29T11:12:44.377 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.387+0000 7f9951a84640 10 monclient: init 2024-03-29T11:12:44.377 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.387+0000 7f9951a84640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:44.377 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.387+0000 7f9951a84640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:12:44.377 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.387+0000 7f9951a84640 10 monclient(hunting): picked mon.noname-a con 0x7f994c0a9d70 addr v1:172.21.15.47:6789/0 2024-03-29T11:12:44.377 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.387+0000 7f9951a84640 10 monclient(hunting): picked mon.noname-c con 0x7f994c13f650 addr v1:172.21.15.47:6790/0 2024-03-29T11:12:44.377 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.387+0000 7f9951a84640 10 monclient(hunting): picked mon.noname-b con 0x7f994c1457f0 addr v1:172.21.15.121:6789/0 2024-03-29T11:12:44.377 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.387+0000 7f9951a84640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f994c13fda0 con 0x7f994c0a9d70 2024-03-29T11:12:44.378 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.387+0000 7f9951a84640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f994c13fff0 con 0x7f994c1457f0 2024-03-29T11:12:44.378 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.387+0000 7f9951a84640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f994c140c20 con 0x7f994c13f650 2024-03-29T11:12:44.378 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.387+0000 7f9951a84640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:44.378 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.387+0000 7f9951a84640 10 monclient(hunting): authenticate will time out at 807.575660s 2024-03-29T11:12:44.378 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.387+0000 7f994affd640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f994c13f650 0x7f994c140510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:54892/0 (socket says 172.21.15.47:54892) 2024-03-29T11:12:44.378 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.387+0000 7f994affd640 1 -- 172.21.15.47:0/3248264701 learned_addr learned my addr 172.21.15.47:0/3248264701 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:12:44.378 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.387+0000 7f9943fff640 10 client.?.objecter ms_handle_connect 0x7f994c13f650 2024-03-29T11:12:44.378 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.387+0000 7f9943fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:44.378 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.387+0000 7f9943fff640 10 client.?.objecter ms_handle_connect 0x7f994c0a9d70 2024-03-29T11:12:44.378 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.387+0000 7f9943fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:44.378 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.387+0000 7f9943fff640 10 client.?.objecter ms_handle_connect 0x7f994c1457f0 2024-03-29T11:12:44.378 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.387+0000 7f9943fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:44.378 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 1 -- 172.21.15.47:0/3248264701 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1769605584 0 0) 0x7f994c140c20 con 0x7f994c13f650 2024-03-29T11:12:44.378 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 10 monclient(hunting): _negotiate 2024-03-29T11:12:44.379 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:44.379 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:44.379 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 10 monclient(hunting): my global_id is 4193 2024-03-29T11:12:44.379 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 1 -- 172.21.15.47:0/3248264701 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f992c004eb0 con 0x7f994c13f650 2024-03-29T11:12:44.379 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 1 -- 172.21.15.47:0/3248264701 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1031378163 0 0) 0x7f994c13fff0 con 0x7f994c1457f0 2024-03-29T11:12:44.379 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 10 monclient(hunting): _negotiate 2024-03-29T11:12:44.379 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:44.379 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:44.379 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 10 monclient(hunting): my global_id is 4189 2024-03-29T11:12:44.379 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 1 -- 172.21.15.47:0/3248264701 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f994c140c20 con 0x7f994c1457f0 2024-03-29T11:12:44.379 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 1 -- 172.21.15.47:0/3248264701 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3141925287 0 0) 0x7f994c13fda0 con 0x7f994c0a9d70 2024-03-29T11:12:44.380 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 10 monclient(hunting): _negotiate 2024-03-29T11:12:44.380 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:44.380 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:44.380 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 10 monclient(hunting): my global_id is 4194 2024-03-29T11:12:44.380 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 1 -- 172.21.15.47:0/3248264701 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f994c13fff0 con 0x7f994c0a9d70 2024-03-29T11:12:44.380 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 1 -- 172.21.15.47:0/3248264701 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2698357056 0 0) 0x7f994c140c20 con 0x7f994c1457f0 2024-03-29T11:12:44.380 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 1 -- 172.21.15.47:0/3248264701 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f994c13fda0 con 0x7f994c1457f0 2024-03-29T11:12:44.380 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 1 -- 172.21.15.47:0/3248264701 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f99440015d0 con 0x7f994c1457f0 2024-03-29T11:12:44.380 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:44.380 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:12:44.380 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 10 monclient(hunting): dump: 2024-03-29T11:12:44.380 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:12:44.380 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:44.381 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:44.381 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:44.381 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:44.381 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:12:44.381 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:44.381 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:44.381 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:44.381 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:12:44.381 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:12:44.381 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 1 -- 172.21.15.47:0/3248264701 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3608841857 0 0) 0x7f992c004eb0 con 0x7f994c13f650 2024-03-29T11:12:44.381 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 1 -- 172.21.15.47:0/3248264701 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f994c140c20 con 0x7f994c13f650 2024-03-29T11:12:44.382 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 1 -- 172.21.15.47:0/3248264701 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f993c001990 con 0x7f994c13f650 2024-03-29T11:12:44.382 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:44.382 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:12:44.382 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 10 monclient(hunting): dump: 2024-03-29T11:12:44.382 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:12:44.382 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:44.382 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:44.382 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:44.382 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:44.382 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:12:44.382 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:44.383 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:44.383 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:44.383 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:12:44.383 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 1 -- 172.21.15.47:0/3248264701 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2465474206 0 0) 0x7f994c13fff0 con 0x7f994c0a9d70 2024-03-29T11:12:44.383 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 1 -- 172.21.15.47:0/3248264701 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f992c004eb0 con 0x7f994c0a9d70 2024-03-29T11:12:44.383 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 1 -- 172.21.15.47:0/3248264701 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f9934001b70 con 0x7f994c0a9d70 2024-03-29T11:12:44.383 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:44.383 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:12:44.383 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 10 monclient(hunting): dump: 2024-03-29T11:12:44.383 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:12:44.383 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:44.383 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:44.383 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:44.383 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:44.384 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:12:44.384 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:44.384 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:44.384 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:44.384 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:12:44.384 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 1 -- 172.21.15.47:0/3248264701 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 864745881 0 0) 0x7f994c13fda0 con 0x7f994c1457f0 2024-03-29T11:12:44.384 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 1 -- 172.21.15.47:0/3248264701 >> v1:172.21.15.47:6790/0 conn(0x7f994c13f650 legacy=0x7f994c140510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:44.384 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 1 -- 172.21.15.47:0/3248264701 >> v1:172.21.15.47:6789/0 conn(0x7f994c0a9d70 legacy=0x7f994c1448f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:44.384 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 10 monclient: _finish_hunting 0 2024-03-29T11:12:44.384 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 1 monclient: found mon.b 2024-03-29T11:12:44.384 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:44.384 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:12:44.384 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 1 -- 172.21.15.47:0/3248264701 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f994c0ab600 con 0x7f994c1457f0 2024-03-29T11:12:44.384 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 10 monclient: _finish_auth 0 2024-03-29T11:12:44.384 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 10 monclient: _check_auth_tickets 2024-03-29T11:12:44.385 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:44.385 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9951a84640 5 monclient: authenticate success, global_id 4189 2024-03-29T11:12:44.385 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9951a84640 10 monclient: _renew_subs 2024-03-29T11:12:44.385 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9951a84640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:12:44.385 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9951a84640 1 -- 172.21.15.47:0/3248264701 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f994c0b0d60 con 0x7f994c1457f0 2024-03-29T11:12:44.385 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9951a84640 10 client.4189.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:12:44.385 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9951a84640 10 monclient: _renew_subs 2024-03-29T11:12:44.385 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9951a84640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:12:44.385 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9951a84640 1 -- 172.21.15.47:0/3248264701 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f994c141e40 con 0x7f994c1457f0 2024-03-29T11:12:44.385 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9951a84640 1 librados: init done 2024-03-29T11:12:44.385 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 1 -- 172.21.15.47:0/3248264701 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9944002030 con 0x7f994c1457f0 2024-03-29T11:12:44.385 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:44.385 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 1 -- 172.21.15.47:0/3248264701 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f9944003b10 con 0x7f994c1457f0 2024-03-29T11:12:44.385 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:44.385 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:44.386 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 10 monclient: dump: 2024-03-29T11:12:44.386 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:12:44.386 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:44.386 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:44.386 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:44.386 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:44.386 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:12:44.386 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:44.386 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:44.386 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:44.386 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:12:44.386 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9949ffb640 4 set_mon_vals no callback set 2024-03-29T11:12:44.386 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 1 -- 172.21.15.47:0/3248264701 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f9944004000 con 0x7f994c1457f0 2024-03-29T11:12:44.386 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 1 -- 172.21.15.47:0/3248264701 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7f99440059b0 con 0x7f994c1457f0 2024-03-29T11:12:44.387 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 10 client.4189.objecter ms_dispatch 0x7f994c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:12:44.387 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 3 client.4189.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:12:44.387 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.388+0000 7f9943fff640 3 client.4189.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:12:44.387 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.389+0000 7f9943fff640 20 client.4189.objecter dump_active .. 0 homeless 2024-03-29T11:12:44.387 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.389+0000 7f9951a84640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:44.387 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.389+0000 7f9951a84640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:44.387 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.389+0000 7f9951a84640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:12:44.387 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.389+0000 7f9951a84640 1 -- 172.21.15.47:0/3248264701 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9930005080 con 0x7f994c1457f0 2024-03-29T11:12:44.387 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.391+0000 7f9943fff640 1 -- 172.21.15.47:0/3248264701 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f99440053f0 con 0x7f994c1457f0 2024-03-29T11:12:44.387 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.391+0000 7f9943fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:44.387 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.391+0000 7f9943fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:12:44.432 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.454+0000 7f9951a84640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:44.432 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.454+0000 7f9951a84640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:44.432 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.454+0000 7f9951a84640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:12:44.432 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.454+0000 7f9951a84640 1 -- 172.21.15.47:0/3248264701 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9930005330 con 0x7f994c1457f0 2024-03-29T11:12:44.433 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.455+0000 7f9943fff640 1 -- 172.21.15.47:0/3248264701 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7f9944005640 con 0x7f994c1457f0 2024-03-29T11:12:44.433 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.455+0000 7f9943fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:44.433 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.455+0000 7f9943fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:12:44.433 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:12:44.433 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:12:44.434 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.456+0000 7f9951a84640 10 librados: watch_flush enter 2024-03-29T11:12:44.435 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.456+0000 7f9951a84640 10 librados: watch_flush exit 2024-03-29T11:12:44.435 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.456+0000 7f9951a84640 20 client.4189.objecter shutdown clearing up homeless session... 2024-03-29T11:12:44.435 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.456+0000 7f9951a84640 10 client.4189.objecter successfully canceled tick 2024-03-29T11:12:44.435 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.456+0000 7f9951a84640 10 monclient: shutdown 2024-03-29T11:12:44.435 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.456+0000 7f9951a84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:44.435 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.456+0000 7f9951a84640 1 -- 172.21.15.47:0/3248264701 >> v1:172.21.15.121:6789/0 conn(0x7f994c1457f0 legacy=0x7f994c0b0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:44.435 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.456+0000 7f9951a84640 1 -- 172.21.15.47:0/3248264701 shutdown_connections 2024-03-29T11:12:44.435 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.456+0000 7f9951a84640 1 -- 172.21.15.47:0/3248264701 >> 172.21.15.47:0/3248264701 conn(0x7f994c066a00 msgr2=0x7f994c13a850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:44.435 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.456+0000 7f9951a84640 1 -- 172.21.15.47:0/3248264701 shutdown_connections 2024-03-29T11:12:44.435 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.456+0000 7f9951a84640 1 -- 172.21.15.47:0/3248264701 wait complete. 2024-03-29T11:12:44.435 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:44.456+0000 7f9951a84640 1 librados: shutdown 2024-03-29T11:12:44.443 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:12:44.797 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-29T11:12:44.819+0000 7f3465a3c040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:12:44.839 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-29T11:12:44.861+0000 7fcab480f040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:12:45.073 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-29T11:12:45.095+0000 7f3465a3c040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:12:45.124 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-29T11:12:45.146+0000 7f3465a3c040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:12:45.124 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-29T11:12:45.146+0000 7fcab480f040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:12:45.177 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-29T11:12:45.199+0000 7fcab480f040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:12:45.240 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-29T11:12:45.262+0000 7f3465a3c040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:12:45.300 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-29T11:12:45.322+0000 7fcab480f040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:12:46.254 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-29T11:12:46.276+0000 7f3406711640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-29T11:12:46.257 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-03-29T11:12:46.279+0000 7f3406711640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-29T11:12:47.328 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-03-29T11:12:47.350+0000 7ffabb413640 -1 osd.4 0 log_to_monitors true 2024-03-29T11:12:47.378 INFO:tasks.ceph.osd.0.smithi047.stderr:2024-03-29T11:12:47.399+0000 7fb3c6d81640 -1 osd.0 0 log_to_monitors true 2024-03-29T11:12:47.379 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-03-29T11:12:47.401+0000 7f4507622640 -1 osd.6 0 log_to_monitors true 2024-03-29T11:12:47.381 INFO:tasks.ceph.osd.3.smithi047.stderr:2024-03-29T11:12:47.402+0000 7ff0f4817640 -1 osd.3 0 log_to_monitors true 2024-03-29T11:12:47.382 INFO:tasks.ceph.osd.2.smithi047.stderr:2024-03-29T11:12:47.404+0000 7f6f05c0d640 -1 osd.2 0 log_to_monitors true 2024-03-29T11:12:47.385 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-03-29T11:12:47.407+0000 7f9464211640 -1 osd.5 0 log_to_monitors true 2024-03-29T11:12:47.393 INFO:tasks.ceph.osd.1.smithi047.stderr:2024-03-29T11:12:47.414+0000 7f45ce4d7640 -1 osd.1 0 log_to_monitors true 2024-03-29T11:12:47.403 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-03-29T11:12:47.425+0000 7f11c4011640 -1 osd.7 0 log_to_monitors true 2024-03-29T11:12:51.250 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:12:51.398 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.420+0000 7ff1e6eff640 10 monclient: get_monmap_and_config 2024-03-29T11:12:51.399 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.420+0000 7ff1e6eff640 10 monclient: build_initial_monmap 2024-03-29T11:12:51.399 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.420+0000 7ff1e6eff640 10 monclient: monmap: 2024-03-29T11:12:51.399 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:12:51.399 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:51.399 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:51.421618+0000 2024-03-29T11:12:51.399 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:51.421618+0000 2024-03-29T11:12:51.399 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:51.399 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:12:51.399 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:12:51.399 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:12:51.399 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:12:51.399 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:12:51.401 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.421+0000 7ff1e6eff640 1 Processor -- start 2024-03-29T11:12:51.401 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.421+0000 7ff1e6eff640 1 -- start start 2024-03-29T11:12:51.401 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.421+0000 7ff1e6eff640 10 monclient: init 2024-03-29T11:12:51.401 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.422+0000 7ff1e6eff640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:51.402 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.422+0000 7ff1e6eff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:12:51.402 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.422+0000 7ff1e6eff640 10 monclient(hunting): picked mon.noname-b con 0x7ff1e01422b0 addr v1:172.21.15.47:6790/0 2024-03-29T11:12:51.402 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.422+0000 7ff1e6eff640 10 monclient(hunting): picked mon.noname-c con 0x7ff1e0142690 addr v1:172.21.15.121:6789/0 2024-03-29T11:12:51.402 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.422+0000 7ff1e6eff640 10 monclient(hunting): picked mon.noname-a con 0x7ff1e00b17d0 addr v1:172.21.15.47:6789/0 2024-03-29T11:12:51.402 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.422+0000 7ff1e6eff640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1e0142c80 con 0x7ff1e00b17d0 2024-03-29T11:12:51.402 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.422+0000 7ff1e6eff640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1e00aae40 con 0x7ff1e0142690 2024-03-29T11:12:51.402 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.422+0000 7ff1e6eff640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1e013fbd0 con 0x7ff1e01422b0 2024-03-29T11:12:51.402 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.422+0000 7ff1e6eff640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:51.402 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.422+0000 7ff1e6eff640 10 monclient(hunting): authenticate will time out at 814.610604s 2024-03-29T11:12:51.402 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.423+0000 7ff1dffff640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7ff1e0142690 0x7ff1e00b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.47:55388/0 (socket says 172.21.15.47:55388) 2024-03-29T11:12:51.402 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.423+0000 7ff1dffff640 1 -- 172.21.15.47:0/2785907298 learned_addr learned my addr 172.21.15.47:0/2785907298 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:12:51.402 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.424+0000 7ff1df7fe640 1 -- 172.21.15.47:0/2785907298 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 718097046 0 0) 0x7ff1e00aae40 con 0x7ff1e0142690 2024-03-29T11:12:51.402 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.424+0000 7ff1df7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:12:51.403 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.424+0000 7ff1df7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:51.403 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.424+0000 7ff1df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:51.403 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.424+0000 7ff1df7fe640 10 monclient(hunting): my global_id is 4231 2024-03-29T11:12:51.403 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.424+0000 7ff1df7fe640 1 -- 172.21.15.47:0/2785907298 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1d0003520 con 0x7ff1e0142690 2024-03-29T11:12:51.411 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.424+0000 7ff1df7fe640 1 -- 172.21.15.47:0/2785907298 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 124988832 0 0) 0x7ff1e013fbd0 con 0x7ff1e01422b0 2024-03-29T11:12:51.411 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.424+0000 7ff1df7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:12:51.412 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.424+0000 7ff1df7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:51.412 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.424+0000 7ff1df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:51.412 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.424+0000 7ff1df7fe640 10 monclient(hunting): my global_id is 4238 2024-03-29T11:12:51.412 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.424+0000 7ff1df7fe640 1 -- 172.21.15.47:0/2785907298 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1e00aae40 con 0x7ff1e01422b0 2024-03-29T11:12:51.412 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.424+0000 7ff1df7fe640 1 -- 172.21.15.47:0/2785907298 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 472011269 0 0) 0x7ff1d0003520 con 0x7ff1e0142690 2024-03-29T11:12:51.412 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.424+0000 7ff1df7fe640 1 -- 172.21.15.47:0/2785907298 >> v1:172.21.15.47:6790/0 conn(0x7ff1e01422b0 legacy=0x7ff1e013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:51.413 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.424+0000 7ff1df7fe640 1 -- 172.21.15.47:0/2785907298 >> v1:172.21.15.47:6789/0 conn(0x7ff1e00b17d0 legacy=0x7ff1e00b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:51.413 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.424+0000 7ff1df7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:12:51.413 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.424+0000 7ff1df7fe640 1 monclient: found mon.noname-c 2024-03-29T11:12:51.413 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.424+0000 7ff1df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:51.413 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.424+0000 7ff1df7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:12:51.413 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.424+0000 7ff1df7fe640 1 -- 172.21.15.47:0/2785907298 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1e0151940 con 0x7ff1e0142690 2024-03-29T11:12:51.413 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.424+0000 7ff1df7fe640 10 monclient: _finish_auth 0 2024-03-29T11:12:51.413 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.424+0000 7ff1df7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:12:51.413 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.424+0000 7ff1df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:51.413 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.424+0000 7ff1df7fe640 1 -- 172.21.15.47:0/2785907298 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7ff1cc001690 con 0x7ff1e0142690 2024-03-29T11:12:51.413 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.424+0000 7ff1df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:51.413 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.424+0000 7ff1df7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:12:51.413 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.424+0000 7ff1df7fe640 10 monclient: dump: 2024-03-29T11:12:51.414 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:12:51.414 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:51.414 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:51.414 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:51.414 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:51.414 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:12:51.414 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:51.414 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:51.414 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:51.414 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:12:51.414 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.424+0000 7ff1e6eff640 5 monclient: authenticate success, global_id 4231 2024-03-29T11:12:51.414 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.424+0000 7ff1e6eff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:51.414 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.425+0000 7ff1df7fe640 1 -- 172.21.15.47:0/2785907298 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1cc0017f0 con 0x7ff1e0142690 2024-03-29T11:12:51.414 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.425+0000 7ff1df7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:51.415 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.425+0000 7ff1e6eff640 10 monclient: get_monmap_and_config success 2024-03-29T11:12:51.415 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.425+0000 7ff1e6eff640 4 set_mon_vals no callback set 2024-03-29T11:12:51.415 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.425+0000 7ff1df7fe640 1 -- 172.21.15.47:0/2785907298 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7ff1cc001bc0 con 0x7ff1e0142690 2024-03-29T11:12:51.415 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.425+0000 7ff1df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:51.415 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.425+0000 7ff1df7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:51.415 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.425+0000 7ff1df7fe640 10 monclient: dump: 2024-03-29T11:12:51.415 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:12:51.415 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:51.415 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:51.415 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:51.415 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:51.415 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:12:51.415 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:51.416 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:51.416 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:51.416 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:12:51.416 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.426+0000 7ff1e6eff640 10 monclient: shutdown 2024-03-29T11:12:51.416 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.426+0000 7ff1e6eff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:51.416 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.426+0000 7ff1e6eff640 1 -- 172.21.15.47:0/2785907298 >> v1:172.21.15.121:6789/0 conn(0x7ff1e0142690 legacy=0x7ff1e00b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:51.416 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.426+0000 7ff1e6eff640 1 -- 172.21.15.47:0/2785907298 shutdown_connections 2024-03-29T11:12:51.416 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.426+0000 7ff1e6eff640 1 -- 172.21.15.47:0/2785907298 >> 172.21.15.47:0/2785907298 conn(0x7ff1e0066a00 msgr2=0x7ff1e0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:51.416 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.426+0000 7ff1e6eff640 1 -- 172.21.15.47:0/2785907298 shutdown_connections 2024-03-29T11:12:51.416 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.426+0000 7ff1e6eff640 1 -- 172.21.15.47:0/2785907298 wait complete. 2024-03-29T11:12:51.416 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.427+0000 7ff1e6eff640 10 monclient: build_initial_monmap 2024-03-29T11:12:51.416 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.427+0000 7ff1e6eff640 10 monclient: monmap: 2024-03-29T11:12:51.416 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:12:51.416 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:12:51.416 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:12:51.417 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:12:51.417 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:51.417 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:12:51.417 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:12:51.417 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:12:51.417 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:12:51.417 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:12:51.417 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.427+0000 7ff1e6eff640 1 librados: starting msgr at 2024-03-29T11:12:51.417 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.427+0000 7ff1e6eff640 1 librados: starting objecter 2024-03-29T11:12:51.417 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.427+0000 7ff1e6eff640 1 Processor -- start 2024-03-29T11:12:51.417 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.428+0000 7ff1e6eff640 1 -- start start 2024-03-29T11:12:51.417 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.428+0000 7ff1e6eff640 1 librados: setting wanted keys 2024-03-29T11:12:51.417 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.428+0000 7ff1e6eff640 1 librados: calling monclient init 2024-03-29T11:12:51.418 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.428+0000 7ff1e6eff640 10 monclient: init 2024-03-29T11:12:51.418 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.429+0000 7ff1e6eff640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:51.418 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.429+0000 7ff1e6eff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:12:51.418 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.429+0000 7ff1e6eff640 10 monclient(hunting): picked mon.noname-b con 0x7ff1e00b17d0 addr v1:172.21.15.121:6789/0 2024-03-29T11:12:51.418 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.429+0000 7ff1e4c9b640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7ff1e00b17d0 0x7ff1e01def10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.47:55536/0 (socket says 172.21.15.47:55536) 2024-03-29T11:12:51.418 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.429+0000 7ff1e6eff640 10 monclient(hunting): picked mon.noname-c con 0x7ff1e01422b0 addr v1:172.21.15.47:6790/0 2024-03-29T11:12:51.418 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.429+0000 7ff1e6eff640 10 monclient(hunting): picked mon.noname-a con 0x7ff1e0142690 addr v1:172.21.15.47:6789/0 2024-03-29T11:12:51.418 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.429+0000 7ff1e4c9b640 1 -- 172.21.15.47:0/2526953923 learned_addr learned my addr 172.21.15.47:0/2526953923 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:12:51.418 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.429+0000 7ff1e6eff640 1 -- 172.21.15.47:0/2526953923 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1e00b65a0 con 0x7ff1e0142690 2024-03-29T11:12:51.418 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.429+0000 7ff1e6eff640 1 -- 172.21.15.47:0/2526953923 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1e012dd60 con 0x7ff1e00b17d0 2024-03-29T11:12:51.418 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.429+0000 7ff1dd7fa640 10 client.?.objecter ms_handle_connect 0x7ff1e01422b0 2024-03-29T11:12:51.418 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.429+0000 7ff1dd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:51.418 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.429+0000 7ff1dd7fa640 10 client.?.objecter ms_handle_connect 0x7ff1e00b17d0 2024-03-29T11:12:51.418 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.429+0000 7ff1dd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:51.419 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.430+0000 7ff1dd7fa640 1 -- 172.21.15.47:0/2526953923 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3213898727 0 0) 0x7ff1e012dd60 con 0x7ff1e00b17d0 2024-03-29T11:12:51.419 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.430+0000 7ff1e6eff640 1 -- 172.21.15.47:0/2526953923 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1e00b6a10 con 0x7ff1e01422b0 2024-03-29T11:12:51.419 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.430+0000 7ff1e6eff640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:51.419 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.430+0000 7ff1e6eff640 10 monclient(hunting): authenticate will time out at 814.618536s 2024-03-29T11:12:51.419 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.430+0000 7ff1dd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:12:51.419 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.430+0000 7ff1dd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:51.419 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.430+0000 7ff1dd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:51.419 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.430+0000 7ff1dd7fa640 10 monclient(hunting): my global_id is 4234 2024-03-29T11:12:51.419 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.430+0000 7ff1dd7fa640 1 -- 172.21.15.47:0/2526953923 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1c4004eb0 con 0x7ff1e00b17d0 2024-03-29T11:12:51.419 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.430+0000 7ff1dd7fa640 10 client.?.objecter ms_handle_connect 0x7ff1e0142690 2024-03-29T11:12:51.419 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.430+0000 7ff1dd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:51.419 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.431+0000 7ff1dd7fa640 1 -- 172.21.15.47:0/2526953923 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1808734892 0 0) 0x7ff1e00b6a10 con 0x7ff1e01422b0 2024-03-29T11:12:51.419 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.431+0000 7ff1dd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:12:51.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.431+0000 7ff1dd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:51.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.431+0000 7ff1dd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:51.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.431+0000 7ff1dd7fa640 10 monclient(hunting): my global_id is 4241 2024-03-29T11:12:51.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.431+0000 7ff1dd7fa640 1 -- 172.21.15.47:0/2526953923 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1e012dd60 con 0x7ff1e01422b0 2024-03-29T11:12:51.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.431+0000 7ff1dd7fa640 1 -- 172.21.15.47:0/2526953923 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2837193799 0 0) 0x7ff1c4004eb0 con 0x7ff1e00b17d0 2024-03-29T11:12:51.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.431+0000 7ff1dd7fa640 1 -- 172.21.15.47:0/2526953923 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff1e00b6a10 con 0x7ff1e00b17d0 2024-03-29T11:12:51.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.431+0000 7ff1dd7fa640 1 -- 172.21.15.47:0/2526953923 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7ff1d40019c0 con 0x7ff1e00b17d0 2024-03-29T11:12:51.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.431+0000 7ff1dd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:51.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.431+0000 7ff1dd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:12:51.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.431+0000 7ff1dd7fa640 10 monclient(hunting): dump: 2024-03-29T11:12:51.420 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:12:51.420 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:51.420 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:51.420 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:51.421 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:51.421 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:12:51.421 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:51.421 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:51.421 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:51.421 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:12:51.421 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.431+0000 7ff1dd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:12:51.421 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.431+0000 7ff1dd7fa640 1 -- 172.21.15.47:0/2526953923 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 887168082 0 0) 0x7ff1e00b6a10 con 0x7ff1e00b17d0 2024-03-29T11:12:51.421 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.432+0000 7ff1dd7fa640 1 -- 172.21.15.47:0/2526953923 >> v1:172.21.15.47:6790/0 conn(0x7ff1e01422b0 legacy=0x7ff1e00b5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:51.421 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.432+0000 7ff1dd7fa640 1 -- 172.21.15.47:0/2526953923 >> v1:172.21.15.47:6789/0 conn(0x7ff1e0142690 legacy=0x7ff1e00b5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:51.421 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.432+0000 7ff1dd7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:12:51.421 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.432+0000 7ff1dd7fa640 1 monclient: found mon.b 2024-03-29T11:12:51.421 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.432+0000 7ff1dd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:51.421 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.432+0000 7ff1dd7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:12:51.421 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.432+0000 7ff1dd7fa640 1 -- 172.21.15.47:0/2526953923 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1e00b6ba0 con 0x7ff1e00b17d0 2024-03-29T11:12:51.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.432+0000 7ff1dd7fa640 10 monclient: _finish_auth 0 2024-03-29T11:12:51.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.432+0000 7ff1dd7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:12:51.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.432+0000 7ff1dd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:51.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.432+0000 7ff1e6eff640 5 monclient: authenticate success, global_id 4234 2024-03-29T11:12:51.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.432+0000 7ff1e6eff640 10 monclient: _renew_subs 2024-03-29T11:12:51.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.432+0000 7ff1e6eff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:12:51.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.432+0000 7ff1e6eff640 1 -- 172.21.15.47:0/2526953923 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1e012ca50 con 0x7ff1e00b17d0 2024-03-29T11:12:51.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.432+0000 7ff1e6eff640 10 client.4234.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:12:51.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.432+0000 7ff1e6eff640 10 monclient: _renew_subs 2024-03-29T11:12:51.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.432+0000 7ff1e6eff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:12:51.423 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.432+0000 7ff1e6eff640 1 -- 172.21.15.47:0/2526953923 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1e01506f0 con 0x7ff1e00b17d0 2024-03-29T11:12:51.423 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.432+0000 7ff1e6eff640 1 librados: init done 2024-03-29T11:12:51.423 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.432+0000 7ff1dd7fa640 1 -- 172.21.15.47:0/2526953923 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1d4003610 con 0x7ff1e00b17d0 2024-03-29T11:12:51.423 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.432+0000 7ff1dd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:51.423 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.432+0000 7ff1dd7fa640 1 -- 172.21.15.47:0/2526953923 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7ff1d40039a0 con 0x7ff1e00b17d0 2024-03-29T11:12:51.423 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.432+0000 7ff1dd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:51.423 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.432+0000 7ff1dd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:51.423 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.432+0000 7ff1dd7fa640 10 monclient: dump: 2024-03-29T11:12:51.423 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:12:51.423 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:51.423 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:51.424 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:51.424 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:51.424 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:12:51.424 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:51.424 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:51.424 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:51.424 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:12:51.425 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.432+0000 7ff1deffd640 4 set_mon_vals no callback set 2024-03-29T11:12:51.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.434+0000 7ff1e6eff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:51.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.434+0000 7ff1e6eff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:51.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.434+0000 7ff1e6eff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:12:51.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.434+0000 7ff1e6eff640 1 -- 172.21.15.47:0/2526953923 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1c0005080 con 0x7ff1e00b17d0 2024-03-29T11:12:51.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.435+0000 7ff1dd7fa640 1 -- 172.21.15.47:0/2526953923 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7ff1d4003b00 con 0x7ff1e00b17d0 2024-03-29T11:12:51.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.436+0000 7ff1dffff640 10 monclient: get_auth_request con 0x7ff1c406dd30 auth_method 0 2024-03-29T11:12:51.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.436+0000 7ff1dd7fa640 1 -- 172.21.15.47:0/2526953923 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7ff1d4084a60 con 0x7ff1e00b17d0 2024-03-29T11:12:51.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.436+0000 7ff1dd7fa640 10 client.4234.objecter ms_dispatch 0x7ff1e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:12:51.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.436+0000 7ff1dd7fa640 3 client.4234.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:12:51.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.436+0000 7ff1dd7fa640 3 client.4234.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:12:51.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.436+0000 7ff1dd7fa640 20 client.4234.objecter dump_active .. 0 homeless 2024-03-29T11:12:51.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.436+0000 7ff1dd7fa640 10 client.4234.objecter ms_handle_connect 0x7ff1c406dd30 2024-03-29T11:12:51.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.442+0000 7ff1dd7fa640 1 -- 172.21.15.47:0/2526953923 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff1d4050340 con 0x7ff1e00b17d0 2024-03-29T11:12:51.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.443+0000 7ff1dd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:51.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.443+0000 7ff1dd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:12:51.849 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.871+0000 7ff1e6eff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:51.849 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.871+0000 7ff1e6eff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:51.849 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.871+0000 7ff1e6eff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:12:51.849 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.871+0000 7ff1e6eff640 1 -- 172.21.15.47:0/2526953923 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff1c0005330 con 0x7ff1e00b17d0 2024-03-29T11:12:51.850 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.872+0000 7ff1dd7fa640 1 -- 172.21.15.47:0/2526953923 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7ff1d4052270 con 0x7ff1e00b17d0 2024-03-29T11:12:51.850 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.872+0000 7ff1dd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:51.850 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.872+0000 7ff1dd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:12:51.850 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:12:51.850 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:12:51.854 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.876+0000 7ff1e6eff640 10 librados: watch_flush enter 2024-03-29T11:12:51.854 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.876+0000 7ff1e6eff640 10 librados: watch_flush exit 2024-03-29T11:12:51.854 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.876+0000 7ff1e6eff640 20 client.4234.objecter shutdown clearing up homeless session... 2024-03-29T11:12:51.854 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.876+0000 7ff1e6eff640 10 client.4234.objecter successfully canceled tick 2024-03-29T11:12:51.854 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.876+0000 7ff1e6eff640 1 -- 172.21.15.47:0/2526953923 >> v1:172.21.15.121:6816/1964117370 conn(0x7ff1c406dd30 legacy=0x7ff1c40701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:51.855 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.876+0000 7ff1e6eff640 10 monclient: shutdown 2024-03-29T11:12:51.855 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.876+0000 7ff1e6eff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:51.855 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.876+0000 7ff1e6eff640 1 -- 172.21.15.47:0/2526953923 >> v1:172.21.15.121:6789/0 conn(0x7ff1e00b17d0 legacy=0x7ff1e01def10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:51.855 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.876+0000 7ff1e6eff640 1 -- 172.21.15.47:0/2526953923 shutdown_connections 2024-03-29T11:12:51.855 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.876+0000 7ff1e6eff640 1 -- 172.21.15.47:0/2526953923 >> 172.21.15.47:0/2526953923 conn(0x7ff1e0066a00 msgr2=0x7ff1e0131da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:51.855 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.877+0000 7ff1e6eff640 1 -- 172.21.15.47:0/2526953923 shutdown_connections 2024-03-29T11:12:51.855 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.877+0000 7ff1e6eff640 1 -- 172.21.15.47:0/2526953923 wait complete. 2024-03-29T11:12:51.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:51.877+0000 7ff1e6eff640 1 librados: shutdown 2024-03-29T11:12:51.873 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:12:58.682 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:12:58.839 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.861+0000 7f7f9830e640 10 monclient: get_monmap_and_config 2024-03-29T11:12:58.839 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.861+0000 7f7f9830e640 10 monclient: build_initial_monmap 2024-03-29T11:12:58.839 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.861+0000 7f7f9830e640 10 monclient: monmap: 2024-03-29T11:12:58.839 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:12:58.839 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:58.839 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:58.862418+0000 2024-03-29T11:12:58.839 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:58.862418+0000 2024-03-29T11:12:58.839 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:58.840 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:12:58.840 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:12:58.840 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:12:58.840 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:12:58.840 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:12:58.841 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.862+0000 7f7f9830e640 1 Processor -- start 2024-03-29T11:12:58.841 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.862+0000 7f7f9830e640 1 -- start start 2024-03-29T11:12:58.841 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.862+0000 7f7f9830e640 10 monclient: init 2024-03-29T11:12:58.841 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.863+0000 7f7f9830e640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:58.841 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.863+0000 7f7f9830e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:12:58.841 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.863+0000 7f7f9830e640 10 monclient(hunting): picked mon.noname-a con 0x7f7f900b4790 addr v1:172.21.15.47:6789/0 2024-03-29T11:12:58.841 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.863+0000 7f7f9830e640 10 monclient(hunting): picked mon.noname-c con 0x7f7f900aea90 addr v1:172.21.15.121:6789/0 2024-03-29T11:12:58.841 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.863+0000 7f7f9830e640 10 monclient(hunting): picked mon.noname-b con 0x7f7f900b39f0 addr v1:172.21.15.47:6790/0 2024-03-29T11:12:58.842 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.863+0000 7f7f9830e640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f900b0c60 con 0x7f7f900b4790 2024-03-29T11:12:58.842 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.863+0000 7f7f9830e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f900af020 con 0x7f7f900aea90 2024-03-29T11:12:58.842 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.863+0000 7f7f9830e640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f9013fbd0 con 0x7f7f900b39f0 2024-03-29T11:12:58.842 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.863+0000 7f7f9830e640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:58.842 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.863+0000 7f7f9830e640 10 monclient(hunting): authenticate will time out at 822.051535s 2024-03-29T11:12:58.845 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.863+0000 7f7f960aa640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f7f900b4790 0x7f7f900b1080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:44184/0 (socket says 172.21.15.47:44184) 2024-03-29T11:12:58.845 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.863+0000 7f7f960aa640 1 -- 172.21.15.47:0/4107179687 learned_addr learned my addr 172.21.15.47:0/4107179687 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:12:58.845 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.864+0000 7f7f950a8640 1 -- 172.21.15.47:0/4107179687 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3634066421 0 0) 0x7f7f900b0c60 con 0x7f7f900b4790 2024-03-29T11:12:58.845 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.864+0000 7f7f950a8640 10 monclient(hunting): _negotiate 2024-03-29T11:12:58.845 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.864+0000 7f7f950a8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:58.845 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.864+0000 7f7f950a8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:58.845 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.864+0000 7f7f950a8640 10 monclient(hunting): my global_id is 4242 2024-03-29T11:12:58.845 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.864+0000 7f7f950a8640 1 -- 172.21.15.47:0/4107179687 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f7c003520 con 0x7f7f900b4790 2024-03-29T11:12:58.846 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.864+0000 7f7f950a8640 1 -- 172.21.15.47:0/4107179687 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1410244644 0 0) 0x7f7f9013fbd0 con 0x7f7f900b39f0 2024-03-29T11:12:58.846 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.864+0000 7f7f950a8640 10 monclient(hunting): _negotiate 2024-03-29T11:12:58.846 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.864+0000 7f7f950a8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:58.846 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.864+0000 7f7f950a8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:58.846 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.864+0000 7f7f950a8640 10 monclient(hunting): my global_id is 4244 2024-03-29T11:12:58.846 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.864+0000 7f7f950a8640 1 -- 172.21.15.47:0/4107179687 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f900b0c60 con 0x7f7f900b39f0 2024-03-29T11:12:58.846 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.865+0000 7f7f950a8640 1 -- 172.21.15.47:0/4107179687 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1096362479 0 0) 0x7f7f7c003520 con 0x7f7f900b4790 2024-03-29T11:12:58.846 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.865+0000 7f7f950a8640 1 -- 172.21.15.47:0/4107179687 >> v1:172.21.15.47:6790/0 conn(0x7f7f900b39f0 legacy=0x7f7f9013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:58.846 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.865+0000 7f7f950a8640 1 -- 172.21.15.47:0/4107179687 >> v1:172.21.15.121:6789/0 conn(0x7f7f900aea90 legacy=0x7f7f900af540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:58.846 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.865+0000 7f7f950a8640 10 monclient: _finish_hunting 0 2024-03-29T11:12:58.846 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.865+0000 7f7f950a8640 1 monclient: found mon.noname-a 2024-03-29T11:12:58.846 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.865+0000 7f7f950a8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:58.846 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.865+0000 7f7f950a8640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-29T11:12:58.846 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.865+0000 7f7f950a8640 1 -- 172.21.15.47:0/4107179687 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f9013fd60 con 0x7f7f900b4790 2024-03-29T11:12:58.847 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.865+0000 7f7f950a8640 10 monclient: _finish_auth 0 2024-03-29T11:12:58.847 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.865+0000 7f7f950a8640 10 monclient: _check_auth_tickets 2024-03-29T11:12:58.847 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.865+0000 7f7f950a8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:58.847 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.865+0000 7f7f950a8640 1 -- 172.21.15.47:0/4107179687 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f7f8c001690 con 0x7f7f900b4790 2024-03-29T11:12:58.847 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.865+0000 7f7f950a8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:58.847 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.865+0000 7f7f950a8640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:12:58.847 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.865+0000 7f7f950a8640 10 monclient: dump: 2024-03-29T11:12:58.847 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:12:58.847 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:58.847 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:58.847 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:58.847 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:58.847 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:12:58.848 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:58.848 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:58.848 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:58.848 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:12:58.848 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.865+0000 7f7f9830e640 5 monclient: authenticate success, global_id 4242 2024-03-29T11:12:58.848 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.865+0000 7f7f9830e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:58.848 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.865+0000 7f7f950a8640 1 -- 172.21.15.47:0/4107179687 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7f8c0017f0 con 0x7f7f900b4790 2024-03-29T11:12:58.848 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.865+0000 7f7f950a8640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:58.848 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.865+0000 7f7f950a8640 1 -- 172.21.15.47:0/4107179687 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f7f8c001c00 con 0x7f7f900b4790 2024-03-29T11:12:58.848 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.865+0000 7f7f950a8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:58.848 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.865+0000 7f7f950a8640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:12:58.848 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.865+0000 7f7f950a8640 10 monclient: dump: 2024-03-29T11:12:58.849 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:12:58.849 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:58.849 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:58.849 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:58.849 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:58.849 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:12:58.849 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:58.849 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:58.849 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:58.849 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:12:58.849 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.865+0000 7f7f9830e640 10 monclient: get_monmap_and_config success 2024-03-29T11:12:58.849 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.865+0000 7f7f9830e640 4 set_mon_vals no callback set 2024-03-29T11:12:58.849 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.866+0000 7f7f9830e640 10 monclient: shutdown 2024-03-29T11:12:58.849 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.866+0000 7f7f9830e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:58.850 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.866+0000 7f7f9830e640 1 -- 172.21.15.47:0/4107179687 >> v1:172.21.15.47:6789/0 conn(0x7f7f900b4790 legacy=0x7f7f900b1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:58.850 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.867+0000 7f7f9830e640 1 -- 172.21.15.47:0/4107179687 shutdown_connections 2024-03-29T11:12:58.850 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.867+0000 7f7f9830e640 1 -- 172.21.15.47:0/4107179687 >> 172.21.15.47:0/4107179687 conn(0x7f7f90066a00 msgr2=0x7f7f90066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:58.850 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.867+0000 7f7f9830e640 1 -- 172.21.15.47:0/4107179687 shutdown_connections 2024-03-29T11:12:58.850 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.867+0000 7f7f9830e640 1 -- 172.21.15.47:0/4107179687 wait complete. 2024-03-29T11:12:58.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.868+0000 7f7f9830e640 10 monclient: build_initial_monmap 2024-03-29T11:12:58.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.868+0000 7f7f9830e640 10 monclient: monmap: 2024-03-29T11:12:58.853 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:12:58.853 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:12:58.854 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:12:58.854 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:12:58.854 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:58.854 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:12:58.854 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:12:58.854 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:12:58.854 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:12:58.854 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:12:58.854 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.869+0000 7f7f9830e640 1 librados: starting msgr at 2024-03-29T11:12:58.854 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.869+0000 7f7f9830e640 1 librados: starting objecter 2024-03-29T11:12:58.854 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.869+0000 7f7f9830e640 1 Processor -- start 2024-03-29T11:12:58.855 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.869+0000 7f7f9830e640 1 -- start start 2024-03-29T11:12:58.855 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.869+0000 7f7f9830e640 1 librados: setting wanted keys 2024-03-29T11:12:58.855 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.869+0000 7f7f9830e640 1 librados: calling monclient init 2024-03-29T11:12:58.855 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.869+0000 7f7f9830e640 10 monclient: init 2024-03-29T11:12:58.855 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.870+0000 7f7f9830e640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:58.855 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.870+0000 7f7f9830e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:12:58.855 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.870+0000 7f7f9830e640 10 monclient(hunting): picked mon.noname-c con 0x7f7f900b39f0 addr v1:172.21.15.47:6790/0 2024-03-29T11:12:58.855 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.870+0000 7f7f9830e640 10 monclient(hunting): picked mon.noname-b con 0x7f7f900b4790 addr v1:172.21.15.121:6789/0 2024-03-29T11:12:58.855 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.870+0000 7f7f9830e640 10 monclient(hunting): picked mon.noname-a con 0x7f7f9006e100 addr v1:172.21.15.47:6789/0 2024-03-29T11:12:58.855 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.870+0000 7f7f9830e640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f9006ef10 con 0x7f7f9006e100 2024-03-29T11:12:58.855 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.870+0000 7f7f9830e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f9006f180 con 0x7f7f900b4790 2024-03-29T11:12:58.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.870+0000 7f7f9830e640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f9006f460 con 0x7f7f900b39f0 2024-03-29T11:12:58.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.870+0000 7f7f9830e640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:58.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.870+0000 7f7f9830e640 10 monclient(hunting): authenticate will time out at 822.058183s 2024-03-29T11:12:58.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.870+0000 7f7f960aa640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f7f900b39f0 0x7f7f90141080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:48326/0 (socket says 172.21.15.47:48326) 2024-03-29T11:12:58.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.870+0000 7f7f960aa640 1 -- 172.21.15.47:0/948575226 learned_addr learned my addr 172.21.15.47:0/948575226 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:12:58.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.870+0000 7f7f86ffd640 10 client.?.objecter ms_handle_connect 0x7f7f900b39f0 2024-03-29T11:12:58.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.870+0000 7f7f86ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:58.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.870+0000 7f7f86ffd640 10 client.?.objecter ms_handle_connect 0x7f7f9006e100 2024-03-29T11:12:58.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.870+0000 7f7f86ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:58.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.870+0000 7f7f86ffd640 10 client.?.objecter ms_handle_connect 0x7f7f900b4790 2024-03-29T11:12:58.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.871+0000 7f7f86ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:58.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.871+0000 7f7f86ffd640 1 -- 172.21.15.47:0/948575226 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 582613010 0 0) 0x7f7f9006f460 con 0x7f7f900b39f0 2024-03-29T11:12:58.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.871+0000 7f7f86ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:12:58.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.871+0000 7f7f86ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:58.857 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.871+0000 7f7f86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:58.857 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.871+0000 7f7f86ffd640 10 monclient(hunting): my global_id is 4247 2024-03-29T11:12:58.857 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.871+0000 7f7f86ffd640 1 -- 172.21.15.47:0/948575226 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f74004eb0 con 0x7f7f900b39f0 2024-03-29T11:12:58.857 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.871+0000 7f7f86ffd640 1 -- 172.21.15.47:0/948575226 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 341209373 0 0) 0x7f7f9006f180 con 0x7f7f900b4790 2024-03-29T11:12:58.857 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.871+0000 7f7f86ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:12:58.857 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.871+0000 7f7f86ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:58.857 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.871+0000 7f7f86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:58.857 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.871+0000 7f7f86ffd640 10 monclient(hunting): my global_id is 4237 2024-03-29T11:12:58.857 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.871+0000 7f7f86ffd640 1 -- 172.21.15.47:0/948575226 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f9006f460 con 0x7f7f900b4790 2024-03-29T11:12:58.857 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.871+0000 7f7f86ffd640 1 -- 172.21.15.47:0/948575226 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4064446945 0 0) 0x7f7f9006ef10 con 0x7f7f9006e100 2024-03-29T11:12:58.857 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.871+0000 7f7f86ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:12:58.857 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.871+0000 7f7f86ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:58.857 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.871+0000 7f7f86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:58.857 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.871+0000 7f7f86ffd640 10 monclient(hunting): my global_id is 4245 2024-03-29T11:12:58.857 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.871+0000 7f7f86ffd640 1 -- 172.21.15.47:0/948575226 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f9006f180 con 0x7f7f9006e100 2024-03-29T11:12:58.858 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.871+0000 7f7f86ffd640 1 -- 172.21.15.47:0/948575226 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1529375901 0 0) 0x7f7f9006f180 con 0x7f7f9006e100 2024-03-29T11:12:58.858 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.871+0000 7f7f86ffd640 1 -- 172.21.15.47:0/948575226 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f9006ef10 con 0x7f7f9006e100 2024-03-29T11:12:58.858 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.871+0000 7f7f86ffd640 1 -- 172.21.15.47:0/948575226 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f7f88001cb0 con 0x7f7f9006e100 2024-03-29T11:12:58.858 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.871+0000 7f7f86ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:58.858 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.871+0000 7f7f86ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:12:58.858 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.871+0000 7f7f86ffd640 10 monclient(hunting): dump: 2024-03-29T11:12:58.858 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:12:58.858 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:58.858 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:58.858 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:58.858 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:58.858 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:12:58.858 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:58.858 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:58.859 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:58.859 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:12:58.859 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.871+0000 7f7f86ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:12:58.859 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.872+0000 7f7f86ffd640 1 -- 172.21.15.47:0/948575226 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2798532139 0 0) 0x7f7f9006ef10 con 0x7f7f9006e100 2024-03-29T11:12:58.859 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.872+0000 7f7f86ffd640 1 -- 172.21.15.47:0/948575226 >> v1:172.21.15.47:6790/0 conn(0x7f7f900b39f0 legacy=0x7f7f90141080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:58.859 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.872+0000 7f7f86ffd640 1 -- 172.21.15.47:0/948575226 >> v1:172.21.15.121:6789/0 conn(0x7f7f900b4790 legacy=0x7f7f9006fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:58.859 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.872+0000 7f7f86ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:12:58.859 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.872+0000 7f7f86ffd640 1 monclient: found mon.a 2024-03-29T11:12:58.859 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.872+0000 7f7f86ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:58.859 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.872+0000 7f7f86ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:12:58.859 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.872+0000 7f7f86ffd640 1 -- 172.21.15.47:0/948575226 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f9006f740 con 0x7f7f9006e100 2024-03-29T11:12:58.859 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.872+0000 7f7f86ffd640 10 monclient: _finish_auth 0 2024-03-29T11:12:58.859 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.872+0000 7f7f86ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:12:58.859 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.872+0000 7f7f86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:58.859 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.872+0000 7f7f9830e640 5 monclient: authenticate success, global_id 4245 2024-03-29T11:12:58.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.872+0000 7f7f9830e640 10 monclient: _renew_subs 2024-03-29T11:12:58.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.872+0000 7f7f9830e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:12:58.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.872+0000 7f7f9830e640 1 -- 172.21.15.47:0/948575226 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f90142460 con 0x7f7f9006e100 2024-03-29T11:12:58.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.872+0000 7f7f9830e640 10 client.4245.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:12:58.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.872+0000 7f7f9830e640 10 monclient: _renew_subs 2024-03-29T11:12:58.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.872+0000 7f7f9830e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:12:58.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.872+0000 7f7f9830e640 1 -- 172.21.15.47:0/948575226 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f90142f70 con 0x7f7f9006e100 2024-03-29T11:12:58.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.872+0000 7f7f9830e640 1 librados: init done 2024-03-29T11:12:58.861 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.872+0000 7f7f86ffd640 1 -- 172.21.15.47:0/948575226 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7f880020c0 con 0x7f7f9006e100 2024-03-29T11:12:58.861 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.872+0000 7f7f86ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:58.861 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.872+0000 7f7f86ffd640 1 -- 172.21.15.47:0/948575226 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f7f88003b10 con 0x7f7f9006e100 2024-03-29T11:12:58.861 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.872+0000 7f7f86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:58.861 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.872+0000 7f7f86ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:12:58.861 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.872+0000 7f7f86ffd640 10 monclient: dump: 2024-03-29T11:12:58.861 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:12:58.861 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:12:58.861 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:58.861 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:12:58.862 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:58.862 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:12:58.862 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:12:58.862 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:12:58.862 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:12:58.862 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:12:58.862 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.872+0000 7f7f87fff640 4 set_mon_vals no callback set 2024-03-29T11:12:58.863 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.873+0000 7f7f9830e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:58.863 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.874+0000 7f7f9830e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:58.863 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.874+0000 7f7f9830e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:12:58.863 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.874+0000 7f7f9830e640 1 -- 172.21.15.47:0/948575226 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f78005080 con 0x7f7f9006e100 2024-03-29T11:12:58.863 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.875+0000 7f7f86ffd640 1 -- 172.21.15.47:0/948575226 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7f7f88003c70 con 0x7f7f9006e100 2024-03-29T11:12:58.864 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.876+0000 7f7f86ffd640 1 -- 172.21.15.47:0/948575226 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7f7f88083af0 con 0x7f7f9006e100 2024-03-29T11:12:58.864 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.876+0000 7f7f86ffd640 10 client.4245.objecter ms_dispatch 0x7f7f90000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:12:58.864 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.876+0000 7f7f86ffd640 3 client.4245.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:12:58.864 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.876+0000 7f7f86ffd640 3 client.4245.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:12:58.864 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.876+0000 7f7f960aa640 10 monclient: get_auth_request con 0x7f7f7406ddb0 auth_method 0 2024-03-29T11:12:58.865 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.876+0000 7f7f86ffd640 20 client.4245.objecter dump_active .. 0 homeless 2024-03-29T11:12:58.865 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.877+0000 7f7f86ffd640 10 client.4245.objecter ms_handle_connect 0x7f7f7406ddb0 2024-03-29T11:12:58.865 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.884+0000 7f7f86ffd640 1 -- 172.21.15.47:0/948575226 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7f8804f3d0 con 0x7f7f9006e100 2024-03-29T11:12:58.865 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.884+0000 7f7f86ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:58.865 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:58.884+0000 7f7f86ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:12:59.205 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:59.227+0000 7f7f9830e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:59.205 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:59.227+0000 7f7f9830e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:59.205 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:59.227+0000 7f7f9830e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:12:59.205 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:59.227+0000 7f7f9830e640 1 -- 172.21.15.47:0/948575226 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7f78005330 con 0x7f7f9006e100 2024-03-29T11:12:59.205 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:59.228+0000 7f7f86ffd640 1 -- 172.21.15.47:0/948575226 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7f7f88051300 con 0x7f7f9006e100 2024-03-29T11:12:59.206 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:59.228+0000 7f7f86ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:59.206 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:59.228+0000 7f7f86ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:12:59.206 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:12:59.206 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:12:59.209 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:59.231+0000 7f7f9830e640 10 librados: watch_flush enter 2024-03-29T11:12:59.209 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:59.231+0000 7f7f9830e640 10 librados: watch_flush exit 2024-03-29T11:12:59.209 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:59.231+0000 7f7f9830e640 20 client.4245.objecter shutdown clearing up homeless session... 2024-03-29T11:12:59.209 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:59.231+0000 7f7f9830e640 10 client.4245.objecter successfully canceled tick 2024-03-29T11:12:59.210 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:59.231+0000 7f7f9830e640 1 -- 172.21.15.47:0/948575226 >> v1:172.21.15.121:6816/1964117370 conn(0x7f7f7406ddb0 legacy=0x7f7f74070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:59.210 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:59.231+0000 7f7f9830e640 10 monclient: shutdown 2024-03-29T11:12:59.210 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:59.231+0000 7f7f9830e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:59.210 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:59.231+0000 7f7f9830e640 1 -- 172.21.15.47:0/948575226 >> v1:172.21.15.47:6789/0 conn(0x7f7f9006e100 legacy=0x7f7f901e60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:59.210 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:59.231+0000 7f7f9830e640 1 -- 172.21.15.47:0/948575226 shutdown_connections 2024-03-29T11:12:59.210 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:59.231+0000 7f7f9830e640 1 -- 172.21.15.47:0/948575226 >> 172.21.15.47:0/948575226 conn(0x7f7f90066a00 msgr2=0x7f7f90135250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:59.210 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:59.232+0000 7f7f9830e640 1 -- 172.21.15.47:0/948575226 shutdown_connections 2024-03-29T11:12:59.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:59.232+0000 7f7f9830e640 1 -- 172.21.15.47:0/948575226 wait complete. 2024-03-29T11:12:59.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:12:59.232+0000 7f7f9830e640 1 librados: shutdown 2024-03-29T11:12:59.224 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:13:06.032 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:13:06.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.198+0000 7f809ccdd640 10 monclient: get_monmap_and_config 2024-03-29T11:13:06.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.198+0000 7f809ccdd640 10 monclient: build_initial_monmap 2024-03-29T11:13:06.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.198+0000 7f809ccdd640 10 monclient: monmap: 2024-03-29T11:13:06.178 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:13:06.178 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:13:06.178 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:13:06.199741+0000 2024-03-29T11:13:06.178 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:13:06.199741+0000 2024-03-29T11:13:06.178 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:06.178 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:13:06.178 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:13:06.178 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:13:06.178 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:13:06.178 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:13:06.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.200+0000 7f809ccdd640 1 Processor -- start 2024-03-29T11:13:06.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.200+0000 7f809ccdd640 1 -- start start 2024-03-29T11:13:06.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.200+0000 7f809ccdd640 10 monclient: init 2024-03-29T11:13:06.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.201+0000 7f809ccdd640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:06.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.201+0000 7f809ccdd640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:13:06.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.201+0000 7f809ccdd640 10 monclient(hunting): picked mon.noname-a con 0x7f80980a9ac0 addr v1:172.21.15.47:6789/0 2024-03-29T11:13:06.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.201+0000 7f809ccdd640 10 monclient(hunting): picked mon.noname-c con 0x7f809813ee90 addr v1:172.21.15.121:6789/0 2024-03-29T11:13:06.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.201+0000 7f809ccdd640 10 monclient(hunting): picked mon.noname-b con 0x7f80980af900 addr v1:172.21.15.47:6790/0 2024-03-29T11:13:06.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.201+0000 7f809ccdd640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80980afd10 con 0x7f80980a9ac0 2024-03-29T11:13:06.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.201+0000 7f809ccdd640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80980af6f0 con 0x7f809813ee90 2024-03-29T11:13:06.180 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.201+0000 7f809ccdd640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80980aac80 con 0x7f80980af900 2024-03-29T11:13:06.180 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.201+0000 7f809ccdd640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:06.180 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.201+0000 7f809ccdd640 10 monclient(hunting): authenticate will time out at 829.389573s 2024-03-29T11:13:06.185 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.202+0000 7f8095d9b640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f809813ee90 0x7f809813b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.47:54738/0 (socket says 172.21.15.47:54738) 2024-03-29T11:13:06.185 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.203+0000 7f8095d9b640 1 -- 172.21.15.47:0/2226474295 learned_addr learned my addr 172.21.15.47:0/2226474295 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:13:06.185 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.204+0000 7f809559a640 1 -- 172.21.15.47:0/2226474295 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1806286160 0 0) 0x7f80980af6f0 con 0x7f809813ee90 2024-03-29T11:13:06.185 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.204+0000 7f809559a640 10 monclient(hunting): _negotiate 2024-03-29T11:13:06.186 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.204+0000 7f809559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:06.186 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.204+0000 7f809559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:06.186 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.204+0000 7f809559a640 10 monclient(hunting): my global_id is 4240 2024-03-29T11:13:06.186 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.204+0000 7f809559a640 1 -- 172.21.15.47:0/2226474295 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8080003500 con 0x7f809813ee90 2024-03-29T11:13:06.186 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.205+0000 7f809559a640 1 -- 172.21.15.47:0/2226474295 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 730967660 0 0) 0x7f8080003500 con 0x7f809813ee90 2024-03-29T11:13:06.186 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.205+0000 7f809559a640 1 -- 172.21.15.47:0/2226474295 >> v1:172.21.15.47:6790/0 conn(0x7f80980af900 legacy=0x7f8098149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:13:06.186 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.205+0000 7f809559a640 1 -- 172.21.15.47:0/2226474295 >> v1:172.21.15.47:6789/0 conn(0x7f80980a9ac0 legacy=0x7f80980adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:13:06.186 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.205+0000 7f809559a640 10 monclient: _finish_hunting 0 2024-03-29T11:13:06.186 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.205+0000 7f809559a640 1 monclient: found mon.noname-c 2024-03-29T11:13:06.186 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.205+0000 7f809559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:06.186 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.205+0000 7f809559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:13:06.186 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.205+0000 7f809559a640 1 -- 172.21.15.47:0/2226474295 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80980aaf60 con 0x7f809813ee90 2024-03-29T11:13:06.186 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.205+0000 7f809559a640 10 monclient: _finish_auth 0 2024-03-29T11:13:06.187 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.205+0000 7f809559a640 10 monclient: _check_auth_tickets 2024-03-29T11:13:06.187 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.205+0000 7f809559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:06.187 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.205+0000 7f809559a640 1 -- 172.21.15.47:0/2226474295 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f8088004cb0 con 0x7f809813ee90 2024-03-29T11:13:06.187 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.205+0000 7f809559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:06.187 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.205+0000 7f809559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:13:06.187 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.205+0000 7f809559a640 10 monclient: dump: 2024-03-29T11:13:06.187 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:13:06.187 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:13:06.187 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:06.187 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:06.187 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:06.187 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:13:06.187 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:13:06.187 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:13:06.187 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:13:06.188 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:13:06.188 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.205+0000 7f809ccdd640 5 monclient: authenticate success, global_id 4240 2024-03-29T11:13:06.188 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.205+0000 7f809ccdd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:13:06.188 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.206+0000 7f809559a640 1 -- 172.21.15.47:0/2226474295 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8088004e80 con 0x7f809813ee90 2024-03-29T11:13:06.188 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.206+0000 7f809559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:06.188 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.206+0000 7f809ccdd640 10 monclient: get_monmap_and_config success 2024-03-29T11:13:06.188 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.206+0000 7f809ccdd640 4 set_mon_vals no callback set 2024-03-29T11:13:06.188 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.207+0000 7f809559a640 1 -- 172.21.15.47:0/2226474295 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f8088001430 con 0x7f809813ee90 2024-03-29T11:13:06.188 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.207+0000 7f809559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:06.188 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.207+0000 7f809559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:13:06.188 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.207+0000 7f809559a640 10 monclient: dump: 2024-03-29T11:13:06.188 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:13:06.188 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:13:06.188 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:06.188 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:06.189 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:06.189 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:13:06.189 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:13:06.189 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:13:06.189 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:13:06.189 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:13:06.189 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.207+0000 7f809ccdd640 10 monclient: shutdown 2024-03-29T11:13:06.189 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.207+0000 7f809ccdd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:06.189 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.207+0000 7f809ccdd640 1 -- 172.21.15.47:0/2226474295 >> v1:172.21.15.121:6789/0 conn(0x7f809813ee90 legacy=0x7f809813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:06.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.207+0000 7f809ccdd640 1 -- 172.21.15.47:0/2226474295 shutdown_connections 2024-03-29T11:13:06.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.207+0000 7f809ccdd640 1 -- 172.21.15.47:0/2226474295 >> 172.21.15.47:0/2226474295 conn(0x7f80980b35e0 msgr2=0x7f80980b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:06.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.208+0000 7f809ccdd640 1 -- 172.21.15.47:0/2226474295 shutdown_connections 2024-03-29T11:13:06.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.208+0000 7f809ccdd640 1 -- 172.21.15.47:0/2226474295 wait complete. 2024-03-29T11:13:06.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.209+0000 7f809ccdd640 10 monclient: build_initial_monmap 2024-03-29T11:13:06.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.209+0000 7f809ccdd640 10 monclient: monmap: 2024-03-29T11:13:06.190 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:13:06.190 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:13:06.190 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:13:06.190 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:13:06.191 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:06.191 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:13:06.191 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:13:06.191 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:13:06.191 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:13:06.191 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:13:06.191 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.209+0000 7f809ccdd640 1 librados: starting msgr at 2024-03-29T11:13:06.191 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.209+0000 7f809ccdd640 1 librados: starting objecter 2024-03-29T11:13:06.191 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.209+0000 7f809ccdd640 1 Processor -- start 2024-03-29T11:13:06.191 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.210+0000 7f809ccdd640 1 -- start start 2024-03-29T11:13:06.191 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.210+0000 7f809ccdd640 1 librados: setting wanted keys 2024-03-29T11:13:06.191 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.210+0000 7f809ccdd640 1 librados: calling monclient init 2024-03-29T11:13:06.191 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.210+0000 7f809ccdd640 10 monclient: init 2024-03-29T11:13:06.191 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.210+0000 7f809ccdd640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:06.191 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.210+0000 7f809ccdd640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:13:06.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.210+0000 7f809ccdd640 10 monclient(hunting): picked mon.noname-b con 0x7f80980a9ac0 addr v1:172.21.15.121:6789/0 2024-03-29T11:13:06.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.214+0000 7f809659c640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f80980a9ac0 0x7f80981dc040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.47:54876/0 (socket says 172.21.15.47:54876) 2024-03-29T11:13:06.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.214+0000 7f809ccdd640 10 monclient(hunting): picked mon.noname-c con 0x7f80980af900 addr v1:172.21.15.47:6790/0 2024-03-29T11:13:06.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.214+0000 7f809ccdd640 10 monclient(hunting): picked mon.noname-a con 0x7f809813ee90 addr v1:172.21.15.47:6789/0 2024-03-29T11:13:06.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.214+0000 7f809ccdd640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8098140d70 con 0x7f809813ee90 2024-03-29T11:13:06.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.214+0000 7f809ccdd640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80981350c0 con 0x7f80980a9ac0 2024-03-29T11:13:06.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.214+0000 7f809ccdd640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80981dbb20 con 0x7f80980af900 2024-03-29T11:13:06.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.214+0000 7f809ccdd640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:06.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.214+0000 7f809ccdd640 10 monclient(hunting): authenticate will time out at 829.402254s 2024-03-29T11:13:06.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.214+0000 7f809659c640 1 -- 172.21.15.47:0/2123184273 learned_addr learned my addr 172.21.15.47:0/2123184273 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:13:06.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.214+0000 7f8086ffd640 10 client.?.objecter ms_handle_connect 0x7f80980a9ac0 2024-03-29T11:13:06.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.214+0000 7f8086ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:06.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.214+0000 7f8086ffd640 1 -- 172.21.15.47:0/2123184273 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 543464266 0 0) 0x7f80981350c0 con 0x7f80980a9ac0 2024-03-29T11:13:06.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.214+0000 7f8086ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:13:06.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.214+0000 7f8086ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:06.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.214+0000 7f8086ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:06.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.214+0000 7f8086ffd640 10 monclient(hunting): my global_id is 4243 2024-03-29T11:13:06.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.214+0000 7f8086ffd640 1 -- 172.21.15.47:0/2123184273 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8078004eb0 con 0x7f80980a9ac0 2024-03-29T11:13:06.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.215+0000 7f8086ffd640 10 client.?.objecter ms_handle_connect 0x7f80980af900 2024-03-29T11:13:06.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.215+0000 7f8086ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:06.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.215+0000 7f8086ffd640 1 -- 172.21.15.47:0/2123184273 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2018334797 0 0) 0x7f8078004eb0 con 0x7f80980a9ac0 2024-03-29T11:13:06.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.215+0000 7f8086ffd640 1 -- 172.21.15.47:0/2123184273 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f80981350c0 con 0x7f80980a9ac0 2024-03-29T11:13:06.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.215+0000 7f8086ffd640 1 -- 172.21.15.47:0/2123184273 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f8090001b50 con 0x7f80980a9ac0 2024-03-29T11:13:06.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.215+0000 7f8086ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:06.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.215+0000 7f8086ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:13:06.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.215+0000 7f8086ffd640 10 monclient(hunting): dump: 2024-03-29T11:13:06.194 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:13:06.194 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:13:06.194 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:06.195 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:06.195 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:06.195 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:13:06.195 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:13:06.195 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:13:06.195 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:13:06.195 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:13:06.195 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.215+0000 7f8086ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:13:06.195 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.215+0000 7f8086ffd640 1 -- 172.21.15.47:0/2123184273 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3440916783 0 0) 0x7f80981dbb20 con 0x7f80980af900 2024-03-29T11:13:06.195 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.215+0000 7f8086ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:13:06.195 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.215+0000 7f8086ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:06.195 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.215+0000 7f8086ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:06.195 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.215+0000 7f8086ffd640 10 monclient(hunting): my global_id is 4250 2024-03-29T11:13:06.195 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.215+0000 7f8086ffd640 1 -- 172.21.15.47:0/2123184273 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8078004eb0 con 0x7f80980af900 2024-03-29T11:13:06.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.215+0000 7f8086ffd640 1 -- 172.21.15.47:0/2123184273 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1835674957 0 0) 0x7f80981350c0 con 0x7f80980a9ac0 2024-03-29T11:13:06.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.215+0000 7f8086ffd640 1 -- 172.21.15.47:0/2123184273 >> v1:172.21.15.47:6790/0 conn(0x7f80980af900 legacy=0x7f80981dacc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:06.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.215+0000 7f8086ffd640 1 -- 172.21.15.47:0/2123184273 >> v1:172.21.15.47:6789/0 conn(0x7f809813ee90 legacy=0x7f80981e71c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:13:06.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.217+0000 7f8086ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:13:06.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.217+0000 7f8086ffd640 1 monclient: found mon.b 2024-03-29T11:13:06.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.217+0000 7f8086ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:06.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.217+0000 7f8086ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:13:06.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.217+0000 7f8086ffd640 1 -- 172.21.15.47:0/2123184273 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8098139640 con 0x7f80980a9ac0 2024-03-29T11:13:06.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.217+0000 7f8086ffd640 10 monclient: _finish_auth 0 2024-03-29T11:13:06.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.217+0000 7f8086ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:13:06.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.217+0000 7f8086ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:06.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.218+0000 7f809ccdd640 5 monclient: authenticate success, global_id 4243 2024-03-29T11:13:06.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.218+0000 7f809ccdd640 10 monclient: _renew_subs 2024-03-29T11:13:06.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.218+0000 7f809ccdd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:13:06.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.218+0000 7f809ccdd640 1 -- 172.21.15.47:0/2123184273 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80981dc750 con 0x7f80980a9ac0 2024-03-29T11:13:06.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.218+0000 7f809ccdd640 10 client.4243.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:13:06.200 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.218+0000 7f809ccdd640 10 monclient: _renew_subs 2024-03-29T11:13:06.200 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.218+0000 7f809ccdd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:13:06.200 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.218+0000 7f809ccdd640 1 -- 172.21.15.47:0/2123184273 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f809813a350 con 0x7f80980a9ac0 2024-03-29T11:13:06.200 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.218+0000 7f809ccdd640 1 librados: init done 2024-03-29T11:13:06.200 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.218+0000 7f8086ffd640 1 -- 172.21.15.47:0/2123184273 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f80900020e0 con 0x7f80980a9ac0 2024-03-29T11:13:06.200 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.218+0000 7f8086ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:06.200 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.219+0000 7f809ccdd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:06.200 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.219+0000 7f809ccdd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:06.200 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.219+0000 7f809ccdd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:13:06.200 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.219+0000 7f8094d99640 4 set_mon_vals no callback set 2024-03-29T11:13:06.200 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.219+0000 7f809ccdd640 1 -- 172.21.15.47:0/2123184273 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80981dbd10 con 0x7f80980a9ac0 2024-03-29T11:13:06.202 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.222+0000 7f8086ffd640 1 -- 172.21.15.47:0/2123184273 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f8090004d80 con 0x7f80980a9ac0 2024-03-29T11:13:06.202 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.222+0000 7f8086ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:06.202 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.222+0000 7f8086ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:13:06.202 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.222+0000 7f8086ffd640 10 monclient: dump: 2024-03-29T11:13:06.202 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:13:06.202 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:13:06.202 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:06.202 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:06.202 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:06.202 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:13:06.202 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:13:06.202 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:13:06.202 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:13:06.203 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:13:06.203 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.223+0000 7f8086ffd640 1 -- 172.21.15.47:0/2123184273 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7f809001add0 con 0x7f80980a9ac0 2024-03-29T11:13:06.203 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.224+0000 7f8086ffd640 1 -- 172.21.15.47:0/2123184273 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7f8090084d40 con 0x7f80980a9ac0 2024-03-29T11:13:06.203 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.224+0000 7f8086ffd640 10 client.4243.objecter ms_dispatch 0x7f8098000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:13:06.203 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.224+0000 7f8086ffd640 3 client.4243.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:13:06.203 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.224+0000 7f8086ffd640 3 client.4243.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:13:06.203 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.224+0000 7f8086ffd640 20 client.4243.objecter dump_active .. 0 homeless 2024-03-29T11:13:06.203 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.224+0000 7f8095d9b640 10 monclient: get_auth_request con 0x7f807806dc10 auth_method 0 2024-03-29T11:13:06.203 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.224+0000 7f8086ffd640 10 client.4243.objecter ms_handle_connect 0x7f807806dc10 2024-03-29T11:13:06.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.229+0000 7f8086ffd640 1 -- 172.21.15.47:0/2123184273 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8090050450 con 0x7f80980a9ac0 2024-03-29T11:13:06.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.229+0000 7f8086ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:06.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.229+0000 7f8086ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:13:06.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.548+0000 7f809ccdd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:06.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.548+0000 7f809ccdd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:06.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.548+0000 7f809ccdd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:13:06.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.548+0000 7f809ccdd640 1 -- 172.21.15.47:0/2123184273 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8098075f10 con 0x7f80980a9ac0 2024-03-29T11:13:06.527 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.549+0000 7f8086ffd640 1 -- 172.21.15.47:0/2123184273 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7f8090052550 con 0x7f80980a9ac0 2024-03-29T11:13:06.527 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.549+0000 7f8086ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:06.527 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.549+0000 7f8086ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:13:06.527 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:13:06.527 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:13:06.531 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.552+0000 7f809ccdd640 10 librados: watch_flush enter 2024-03-29T11:13:06.531 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.552+0000 7f809ccdd640 10 librados: watch_flush exit 2024-03-29T11:13:06.531 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.552+0000 7f809ccdd640 20 client.4243.objecter shutdown clearing up homeless session... 2024-03-29T11:13:06.531 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.552+0000 7f809ccdd640 10 client.4243.objecter successfully canceled tick 2024-03-29T11:13:06.531 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.552+0000 7f809ccdd640 1 -- 172.21.15.47:0/2123184273 >> v1:172.21.15.121:6816/1964117370 conn(0x7f807806dc10 legacy=0x7f80780700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:06.531 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.552+0000 7f809ccdd640 10 monclient: shutdown 2024-03-29T11:13:06.531 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.552+0000 7f809ccdd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:06.531 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.552+0000 7f809ccdd640 1 -- 172.21.15.47:0/2123184273 >> v1:172.21.15.121:6789/0 conn(0x7f80980a9ac0 legacy=0x7f80981dc040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:06.531 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.552+0000 7f809ccdd640 1 -- 172.21.15.47:0/2123184273 shutdown_connections 2024-03-29T11:13:06.531 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.552+0000 7f809ccdd640 1 -- 172.21.15.47:0/2123184273 >> 172.21.15.47:0/2123184273 conn(0x7f80980b35e0 msgr2=0x7f809813b190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:06.531 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.553+0000 7f809ccdd640 1 -- 172.21.15.47:0/2123184273 shutdown_connections 2024-03-29T11:13:06.531 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.553+0000 7f809ccdd640 1 -- 172.21.15.47:0/2123184273 wait complete. 2024-03-29T11:13:06.531 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:06.553+0000 7f809ccdd640 1 librados: shutdown 2024-03-29T11:13:06.549 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:13:13.358 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:13:13.511 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.533+0000 7f8425e40640 10 monclient: get_monmap_and_config 2024-03-29T11:13:13.511 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.533+0000 7f8425e40640 10 monclient: build_initial_monmap 2024-03-29T11:13:13.511 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.533+0000 7f8425e40640 10 monclient: monmap: 2024-03-29T11:13:13.512 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:13:13.512 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:13:13.512 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:13:13.534633+0000 2024-03-29T11:13:13.512 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:13:13.534633+0000 2024-03-29T11:13:13.512 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:13.512 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:13:13.512 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:13:13.512 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:13:13.512 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:13:13.512 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:13:13.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.534+0000 7f8425e40640 1 Processor -- start 2024-03-29T11:13:13.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.534+0000 7f8425e40640 1 -- start start 2024-03-29T11:13:13.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.534+0000 7f8425e40640 10 monclient: init 2024-03-29T11:13:13.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.535+0000 7f8425e40640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:13.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.535+0000 7f8425e40640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:13:13.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.535+0000 7f8425e40640 10 monclient(hunting): picked mon.noname-c con 0x7f84201457f0 addr v1:172.21.15.121:6789/0 2024-03-29T11:13:13.517 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.535+0000 7f8425e40640 10 monclient(hunting): picked mon.noname-b con 0x7f842013f650 addr v1:172.21.15.47:6790/0 2024-03-29T11:13:13.517 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.535+0000 7f8425e40640 10 monclient(hunting): picked mon.noname-a con 0x7f84200a9d70 addr v1:172.21.15.47:6789/0 2024-03-29T11:13:13.517 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.535+0000 7f8425e40640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84200aaef0 con 0x7f84200a9d70 2024-03-29T11:13:13.517 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.535+0000 7f8425e40640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8420145bd0 con 0x7f84201457f0 2024-03-29T11:13:13.517 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.535+0000 7f8425e40640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8420145eb0 con 0x7f842013f650 2024-03-29T11:13:13.517 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.535+0000 7f8425e40640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:13.518 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.535+0000 7f8425e40640 10 monclient(hunting): authenticate will time out at 836.723764s 2024-03-29T11:13:13.518 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.537+0000 7f841ffff640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f84200a9d70 0x7f842014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:50133/0 (socket says 172.21.15.47:50133) 2024-03-29T11:13:13.518 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.537+0000 7f841ffff640 1 -- 172.21.15.47:0/422123855 learned_addr learned my addr 172.21.15.47:0/422123855 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:13:13.518 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.538+0000 7f841e7fc640 1 -- 172.21.15.47:0/422123855 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 469177777 0 0) 0x7f84200aaef0 con 0x7f84200a9d70 2024-03-29T11:13:13.518 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.538+0000 7f841e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:13:13.518 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.538+0000 7f841e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:13.518 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.538+0000 7f841e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:13.518 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.538+0000 7f841e7fc640 10 monclient(hunting): my global_id is 4248 2024-03-29T11:13:13.518 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.538+0000 7f841e7fc640 1 -- 172.21.15.47:0/422123855 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f840c003520 con 0x7f84200a9d70 2024-03-29T11:13:13.518 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.538+0000 7f841e7fc640 1 -- 172.21.15.47:0/422123855 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2524370030 0 0) 0x7f840c003520 con 0x7f84200a9d70 2024-03-29T11:13:13.518 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.538+0000 7f841e7fc640 1 -- 172.21.15.47:0/422123855 >> v1:172.21.15.47:6790/0 conn(0x7f842013f650 legacy=0x7f84200b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:13:13.518 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.538+0000 7f841e7fc640 1 -- 172.21.15.47:0/422123855 >> v1:172.21.15.121:6789/0 conn(0x7f84201457f0 legacy=0x7f842013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:13:13.518 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.538+0000 7f841e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:13:13.518 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.538+0000 7f841e7fc640 1 monclient: found mon.noname-a 2024-03-29T11:13:13.519 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.538+0000 7f841e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:13.519 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.538+0000 7f841e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-29T11:13:13.519 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.538+0000 7f841e7fc640 1 -- 172.21.15.47:0/422123855 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8420146340 con 0x7f84200a9d70 2024-03-29T11:13:13.519 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.538+0000 7f841e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:13:13.519 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.538+0000 7f841e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:13:13.519 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.538+0000 7f841e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:13.519 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.538+0000 7f841e7fc640 1 -- 172.21.15.47:0/422123855 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f8418001a10 con 0x7f84200a9d70 2024-03-29T11:13:13.519 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.538+0000 7f841e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:13.519 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.538+0000 7f841e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:13:13.519 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.538+0000 7f841e7fc640 10 monclient: dump: 2024-03-29T11:13:13.519 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:13:13.519 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:13:13.519 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:13.519 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:13.519 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:13.520 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:13:13.520 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:13:13.520 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:13:13.520 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:13:13.520 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:13:13.520 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.539+0000 7f8425e40640 5 monclient: authenticate success, global_id 4248 2024-03-29T11:13:13.520 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.539+0000 7f8425e40640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:13:13.520 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.539+0000 7f841e7fc640 1 -- 172.21.15.47:0/422123855 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8418001be0 con 0x7f84200a9d70 2024-03-29T11:13:13.527 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.539+0000 7f841e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:13.528 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.539+0000 7f841e7fc640 1 -- 172.21.15.47:0/422123855 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f8418002030 con 0x7f84200a9d70 2024-03-29T11:13:13.528 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.539+0000 7f841e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:13.528 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.539+0000 7f841e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:13:13.528 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.539+0000 7f841e7fc640 10 monclient: dump: 2024-03-29T11:13:13.528 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:13:13.528 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:13:13.528 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:13.528 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:13.528 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:13.529 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:13:13.529 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:13:13.529 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:13:13.529 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:13:13.529 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:13:13.529 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.539+0000 7f8425e40640 10 monclient: get_monmap_and_config success 2024-03-29T11:13:13.529 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.539+0000 7f8425e40640 4 set_mon_vals no callback set 2024-03-29T11:13:13.529 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.540+0000 7f8425e40640 10 monclient: shutdown 2024-03-29T11:13:13.529 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.540+0000 7f8425e40640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:13.529 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.540+0000 7f8425e40640 1 -- 172.21.15.47:0/422123855 >> v1:172.21.15.47:6789/0 conn(0x7f84200a9d70 legacy=0x7f842014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:13.529 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.541+0000 7f8425e40640 1 -- 172.21.15.47:0/422123855 shutdown_connections 2024-03-29T11:13:13.529 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.541+0000 7f8425e40640 1 -- 172.21.15.47:0/422123855 >> 172.21.15.47:0/422123855 conn(0x7f8420066a00 msgr2=0x7f8420066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:13.529 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.542+0000 7f8425e40640 1 -- 172.21.15.47:0/422123855 shutdown_connections 2024-03-29T11:13:13.529 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.542+0000 7f8425e40640 1 -- 172.21.15.47:0/422123855 wait complete. 2024-03-29T11:13:13.529 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.543+0000 7f8425e40640 10 monclient: build_initial_monmap 2024-03-29T11:13:13.530 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.543+0000 7f8425e40640 10 monclient: monmap: 2024-03-29T11:13:13.530 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:13:13.530 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:13:13.530 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:13:13.530 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:13:13.530 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:13.530 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:13:13.530 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:13:13.530 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:13:13.530 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:13:13.530 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:13:13.530 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.543+0000 7f8425e40640 1 librados: starting msgr at 2024-03-29T11:13:13.530 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.543+0000 7f8425e40640 1 librados: starting objecter 2024-03-29T11:13:13.530 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.543+0000 7f8425e40640 1 Processor -- start 2024-03-29T11:13:13.530 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.544+0000 7f8425e40640 1 -- start start 2024-03-29T11:13:13.531 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.544+0000 7f8425e40640 1 librados: setting wanted keys 2024-03-29T11:13:13.531 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.544+0000 7f8425e40640 1 librados: calling monclient init 2024-03-29T11:13:13.531 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.544+0000 7f8425e40640 10 monclient: init 2024-03-29T11:13:13.531 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.544+0000 7f8425e40640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:13.532 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.544+0000 7f8425e40640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:13:13.532 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.544+0000 7f8425e40640 10 monclient(hunting): picked mon.noname-b con 0x7f84200a9d70 addr v1:172.21.15.121:6789/0 2024-03-29T11:13:13.532 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.544+0000 7f8425e40640 10 monclient(hunting): picked mon.noname-c con 0x7f842013f650 addr v1:172.21.15.47:6790/0 2024-03-29T11:13:13.532 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.544+0000 7f8425e40640 10 monclient(hunting): picked mon.noname-a con 0x7f84201457f0 addr v1:172.21.15.47:6789/0 2024-03-29T11:13:13.532 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.544+0000 7f8425e40640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f842006f1a0 con 0x7f84201457f0 2024-03-29T11:13:13.532 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.544+0000 7f8425e40640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f842006f330 con 0x7f84200a9d70 2024-03-29T11:13:13.532 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.544+0000 7f8425e40640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8420137000 con 0x7f842013f650 2024-03-29T11:13:13.532 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.544+0000 7f8425e40640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:13.532 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.544+0000 7f8425e40640 10 monclient(hunting): authenticate will time out at 836.732859s 2024-03-29T11:13:13.536 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.547+0000 7f841ffff640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f84201457f0 0x7f84201e6540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:50231/0 (socket says 172.21.15.47:50231) 2024-03-29T11:13:13.536 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.547+0000 7f841ffff640 1 -- 172.21.15.47:0/3667642594 learned_addr learned my addr 172.21.15.47:0/3667642594 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:13:13.536 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.548+0000 7f8417fff640 10 client.?.objecter ms_handle_connect 0x7f84201457f0 2024-03-29T11:13:13.536 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.548+0000 7f8417fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:13.536 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.548+0000 7f8417fff640 10 client.?.objecter ms_handle_connect 0x7f84200a9d70 2024-03-29T11:13:13.536 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.548+0000 7f8417fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:13.536 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.548+0000 7f8417fff640 1 -- 172.21.15.47:0/3667642594 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2757510272 0 0) 0x7f842006f1a0 con 0x7f84201457f0 2024-03-29T11:13:13.536 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.549+0000 7f8417fff640 10 monclient(hunting): _negotiate 2024-03-29T11:13:13.536 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.549+0000 7f8417fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:13.536 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.549+0000 7f8417fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:13.537 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.549+0000 7f8417fff640 10 monclient(hunting): my global_id is 4251 2024-03-29T11:13:13.537 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.549+0000 7f8417fff640 1 -- 172.21.15.47:0/3667642594 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8400004eb0 con 0x7f84201457f0 2024-03-29T11:13:13.537 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.549+0000 7f8417fff640 1 -- 172.21.15.47:0/3667642594 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2234371223 0 0) 0x7f8400004eb0 con 0x7f84201457f0 2024-03-29T11:13:13.537 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.549+0000 7f8417fff640 1 -- 172.21.15.47:0/3667642594 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f842006f1a0 con 0x7f84201457f0 2024-03-29T11:13:13.537 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.549+0000 7f8417fff640 1 -- 172.21.15.47:0/3667642594 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f8418001ef0 con 0x7f84201457f0 2024-03-29T11:13:13.537 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.549+0000 7f8417fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:13.537 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.549+0000 7f8417fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:13:13.537 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.549+0000 7f8417fff640 10 monclient(hunting): dump: 2024-03-29T11:13:13.537 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:13:13.538 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:13:13.538 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:13.538 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:13.538 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:13.538 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:13:13.538 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:13:13.538 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:13:13.538 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:13:13.538 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:13:13.538 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.549+0000 7f8417fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:13:13.538 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.549+0000 7f8417fff640 1 -- 172.21.15.47:0/3667642594 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2726962196 0 0) 0x7f842006f330 con 0x7f84200a9d70 2024-03-29T11:13:13.538 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.549+0000 7f8417fff640 10 monclient(hunting): _negotiate 2024-03-29T11:13:13.538 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.549+0000 7f8417fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:13.538 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.549+0000 7f8417fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:13.539 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.549+0000 7f8417fff640 10 monclient(hunting): my global_id is 4246 2024-03-29T11:13:13.539 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.549+0000 7f8417fff640 1 -- 172.21.15.47:0/3667642594 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8400004eb0 con 0x7f84200a9d70 2024-03-29T11:13:13.539 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.549+0000 7f8417fff640 1 -- 172.21.15.47:0/3667642594 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1538567176 0 0) 0x7f842006f1a0 con 0x7f84201457f0 2024-03-29T11:13:13.539 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.549+0000 7f8417fff640 1 -- 172.21.15.47:0/3667642594 >> v1:172.21.15.47:6790/0 conn(0x7f842013f650 legacy=0x7f84201e2e10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:13:13.539 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.550+0000 7f8417fff640 1 -- 172.21.15.47:0/3667642594 >> v1:172.21.15.121:6789/0 conn(0x7f84200a9d70 legacy=0x7f84200708f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:13.539 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.550+0000 7f8417fff640 10 monclient: _finish_hunting 0 2024-03-29T11:13:13.539 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.550+0000 7f8417fff640 1 monclient: found mon.a 2024-03-29T11:13:13.539 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.550+0000 7f8417fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:13.539 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.550+0000 7f8417fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:13:13.539 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.550+0000 7f8417fff640 1 -- 172.21.15.47:0/3667642594 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8420137190 con 0x7f84201457f0 2024-03-29T11:13:13.539 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.551+0000 7f8417fff640 10 monclient: _finish_auth 0 2024-03-29T11:13:13.539 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.551+0000 7f8417fff640 10 monclient: _check_auth_tickets 2024-03-29T11:13:13.539 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.551+0000 7f8417fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:13.539 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.551+0000 7f8425e40640 5 monclient: authenticate success, global_id 4251 2024-03-29T11:13:13.539 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.551+0000 7f8425e40640 10 monclient: _renew_subs 2024-03-29T11:13:13.540 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.551+0000 7f8425e40640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:13:13.540 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.551+0000 7f8425e40640 1 -- 172.21.15.47:0/3667642594 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8420071000 con 0x7f84201457f0 2024-03-29T11:13:13.540 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.551+0000 7f8425e40640 10 client.4251.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:13:13.540 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.551+0000 7f8425e40640 10 monclient: _renew_subs 2024-03-29T11:13:13.540 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.551+0000 7f8425e40640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:13:13.540 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.551+0000 7f8425e40640 1 -- 172.21.15.47:0/3667642594 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8420070110 con 0x7f84201457f0 2024-03-29T11:13:13.540 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.551+0000 7f8425e40640 1 librados: init done 2024-03-29T11:13:13.540 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.551+0000 7f8417fff640 1 -- 172.21.15.47:0/3667642594 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8418004f70 con 0x7f84201457f0 2024-03-29T11:13:13.540 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.551+0000 7f8417fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:13.540 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.551+0000 7f841dffb640 4 set_mon_vals no callback set 2024-03-29T11:13:13.540 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.552+0000 7f8417fff640 1 -- 172.21.15.47:0/3667642594 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f8418005280 con 0x7f84201457f0 2024-03-29T11:13:13.540 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.552+0000 7f8417fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:13.540 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.552+0000 7f8417fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:13:13.540 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.552+0000 7f8417fff640 10 monclient: dump: 2024-03-29T11:13:13.540 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:13:13.541 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:13:13.541 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:13.541 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:13.541 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:13.541 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:13:13.541 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:13:13.541 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:13:13.541 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:13:13.541 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:13:13.541 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.552+0000 7f8425e40640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:13.541 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.552+0000 7f8425e40640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:13.541 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.552+0000 7f8425e40640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:13:13.541 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.552+0000 7f8425e40640 1 -- 172.21.15.47:0/3667642594 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8404005080 con 0x7f84201457f0 2024-03-29T11:13:13.541 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.555+0000 7f8417fff640 1 -- 172.21.15.47:0/3667642594 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7f841801b250 con 0x7f84201457f0 2024-03-29T11:13:13.541 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.555+0000 7f8417fff640 1 -- 172.21.15.47:0/3667642594 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7f8418084f70 con 0x7f84201457f0 2024-03-29T11:13:13.542 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.555+0000 7f8417fff640 10 client.4251.objecter ms_dispatch 0x7f8420000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:13:13.542 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.555+0000 7f8417fff640 3 client.4251.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:13:13.542 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.555+0000 7f8417fff640 3 client.4251.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:13:13.542 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.555+0000 7f8417fff640 20 client.4251.objecter dump_active .. 0 homeless 2024-03-29T11:13:13.542 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.556+0000 7f841f7fe640 10 monclient: get_auth_request con 0x7f840006dd10 auth_method 0 2024-03-29T11:13:13.542 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.556+0000 7f8417fff640 10 client.4251.objecter ms_handle_connect 0x7f840006dd10 2024-03-29T11:13:13.543 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.563+0000 7f8417fff640 1 -- 172.21.15.47:0/3667642594 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8418050850 con 0x7f84201457f0 2024-03-29T11:13:13.543 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.563+0000 7f8417fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:13.543 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.563+0000 7f8417fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:13:13.886 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.909+0000 7f8425e40640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:13.887 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.909+0000 7f8425e40640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:13.887 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.909+0000 7f8425e40640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:13:13.887 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.909+0000 7f8425e40640 1 -- 172.21.15.47:0/3667642594 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8404005330 con 0x7f84201457f0 2024-03-29T11:13:13.889 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.912+0000 7f8417fff640 1 -- 172.21.15.47:0/3667642594 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7f8418052780 con 0x7f84201457f0 2024-03-29T11:13:13.890 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.912+0000 7f8417fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:13.890 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.912+0000 7f8417fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:13:13.890 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:13:13.890 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:13:13.895 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.915+0000 7f8425e40640 10 librados: watch_flush enter 2024-03-29T11:13:13.895 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.915+0000 7f8425e40640 10 librados: watch_flush exit 2024-03-29T11:13:13.895 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.915+0000 7f8425e40640 20 client.4251.objecter shutdown clearing up homeless session... 2024-03-29T11:13:13.895 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.915+0000 7f8425e40640 10 client.4251.objecter successfully canceled tick 2024-03-29T11:13:13.895 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.915+0000 7f8425e40640 1 -- 172.21.15.47:0/3667642594 >> v1:172.21.15.121:6816/1964117370 conn(0x7f840006dd10 legacy=0x7f84000701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:13.895 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.915+0000 7f8425e40640 10 monclient: shutdown 2024-03-29T11:13:13.896 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.915+0000 7f8425e40640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:13.896 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.915+0000 7f8425e40640 1 -- 172.21.15.47:0/3667642594 >> v1:172.21.15.47:6789/0 conn(0x7f84201457f0 legacy=0x7f84201e6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:13.896 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.915+0000 7f8425e40640 1 -- 172.21.15.47:0/3667642594 shutdown_connections 2024-03-29T11:13:13.896 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.915+0000 7f8425e40640 1 -- 172.21.15.47:0/3667642594 >> 172.21.15.47:0/3667642594 conn(0x7f8420066a00 msgr2=0x7f84200b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:13.896 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.916+0000 7f8425e40640 1 -- 172.21.15.47:0/3667642594 shutdown_connections 2024-03-29T11:13:13.896 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.916+0000 7f8425e40640 1 -- 172.21.15.47:0/3667642594 wait complete. 2024-03-29T11:13:13.896 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:13.916+0000 7f8425e40640 1 librados: shutdown 2024-03-29T11:13:13.910 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:13:20.717 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:13:20.867 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.888+0000 7faa3a5dc640 10 monclient: get_monmap_and_config 2024-03-29T11:13:20.867 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.888+0000 7faa3a5dc640 10 monclient: build_initial_monmap 2024-03-29T11:13:20.868 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.888+0000 7faa3a5dc640 10 monclient: monmap: 2024-03-29T11:13:20.868 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:13:20.868 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:13:20.868 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:13:20.889681+0000 2024-03-29T11:13:20.868 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:13:20.889681+0000 2024-03-29T11:13:20.868 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:20.868 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:13:20.868 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:13:20.868 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:13:20.868 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:13:20.868 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:13:20.868 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.889+0000 7faa3a5dc640 1 Processor -- start 2024-03-29T11:13:20.869 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.890+0000 7faa3a5dc640 1 -- start start 2024-03-29T11:13:20.869 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.890+0000 7faa3a5dc640 10 monclient: init 2024-03-29T11:13:20.869 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.891+0000 7faa3a5dc640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:20.870 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.891+0000 7faa3a5dc640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:13:20.870 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.891+0000 7faa3a5dc640 10 monclient(hunting): picked mon.noname-c con 0x7faa340b3f30 addr v1:172.21.15.121:6789/0 2024-03-29T11:13:20.870 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.891+0000 7faa3a5dc640 10 monclient(hunting): picked mon.noname-a con 0x7faa340ae220 addr v1:172.21.15.47:6789/0 2024-03-29T11:13:20.870 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.891+0000 7faa3a5dc640 10 monclient(hunting): picked mon.noname-b con 0x7faa3413b9d0 addr v1:172.21.15.47:6790/0 2024-03-29T11:13:20.870 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.891+0000 7faa3a5dc640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa34142490 con 0x7faa340ae220 2024-03-29T11:13:20.870 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.891+0000 7faa3a5dc640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa340ac130 con 0x7faa340b3f30 2024-03-29T11:13:20.870 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.891+0000 7faa3a5dc640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa340ac410 con 0x7faa3413b9d0 2024-03-29T11:13:20.870 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.891+0000 7faa3a5dc640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:20.870 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.891+0000 7faa3a5dc640 10 monclient(hunting): authenticate will time out at 844.079248s 2024-03-29T11:13:20.872 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.894+0000 7faa395da640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7faa340b3f30 0x7faa34141d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.47:53321/0 (socket says 172.21.15.47:53321) 2024-03-29T11:13:20.872 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.894+0000 7faa395da640 1 -- 172.21.15.47:0/1633365370 learned_addr learned my addr 172.21.15.47:0/1633365370 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:13:20.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.895+0000 7faa2bfff640 1 -- 172.21.15.47:0/1633365370 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 425009948 0 0) 0x7faa340ac130 con 0x7faa340b3f30 2024-03-29T11:13:20.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.895+0000 7faa2bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:13:20.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.895+0000 7faa2bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:20.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.895+0000 7faa2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:20.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.895+0000 7faa2bfff640 10 monclient(hunting): my global_id is 4249 2024-03-29T11:13:20.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.895+0000 7faa2bfff640 1 -- 172.21.15.47:0/1633365370 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa20003520 con 0x7faa340b3f30 2024-03-29T11:13:20.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.895+0000 7faa2bfff640 1 -- 172.21.15.47:0/1633365370 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1336368419 0 0) 0x7faa340ac410 con 0x7faa3413b9d0 2024-03-29T11:13:20.880 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.895+0000 7faa2bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:13:20.880 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.895+0000 7faa2bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:20.880 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.895+0000 7faa2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:20.880 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.895+0000 7faa2bfff640 10 monclient(hunting): my global_id is 4253 2024-03-29T11:13:20.880 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.895+0000 7faa2bfff640 1 -- 172.21.15.47:0/1633365370 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa340ac130 con 0x7faa3413b9d0 2024-03-29T11:13:20.880 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.896+0000 7faa2bfff640 1 -- 172.21.15.47:0/1633365370 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2120664309 0 0) 0x7faa340ac130 con 0x7faa3413b9d0 2024-03-29T11:13:20.880 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.896+0000 7faa2bfff640 1 -- 172.21.15.47:0/1633365370 >> v1:172.21.15.121:6789/0 conn(0x7faa340b3f30 legacy=0x7faa34141d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:20.880 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.896+0000 7faa2bfff640 1 -- 172.21.15.47:0/1633365370 >> v1:172.21.15.47:6789/0 conn(0x7faa340ae220 legacy=0x7faa3413b2c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:13:20.880 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.897+0000 7faa2bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:13:20.880 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.897+0000 7faa2bfff640 1 monclient: found mon.noname-b 2024-03-29T11:13:20.880 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.897+0000 7faa2bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:20.880 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.897+0000 7faa2bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-29T11:13:20.881 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.897+0000 7faa2bfff640 1 -- 172.21.15.47:0/1633365370 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa340afa40 con 0x7faa3413b9d0 2024-03-29T11:13:20.881 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.897+0000 7faa2bfff640 10 monclient: _finish_auth 0 2024-03-29T11:13:20.881 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.897+0000 7faa2bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:13:20.881 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.897+0000 7faa2bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:20.881 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.898+0000 7faa3a5dc640 5 monclient: authenticate success, global_id 4253 2024-03-29T11:13:20.881 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.898+0000 7faa3a5dc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:13:20.881 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.898+0000 7faa2bfff640 1 -- 172.21.15.47:0/1633365370 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7faa24001670 con 0x7faa3413b9d0 2024-03-29T11:13:20.881 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.898+0000 7faa2bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:20.881 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.898+0000 7faa2bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:13:20.881 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.898+0000 7faa2bfff640 10 monclient: dump: 2024-03-29T11:13:20.881 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:13:20.882 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:13:20.882 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:20.882 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:20.882 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:20.882 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:13:20.882 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:13:20.882 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:13:20.882 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:13:20.882 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:13:20.882 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.898+0000 7faa3a5dc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:13:20.882 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.898+0000 7faa2bfff640 1 -- 172.21.15.47:0/1633365370 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa24001890 con 0x7faa3413b9d0 2024-03-29T11:13:20.882 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.898+0000 7faa2bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:20.883 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.898+0000 7faa3a5dc640 10 monclient: get_monmap_and_config success 2024-03-29T11:13:20.883 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.898+0000 7faa3a5dc640 4 set_mon_vals no callback set 2024-03-29T11:13:20.883 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.899+0000 7faa3a5dc640 10 monclient: shutdown 2024-03-29T11:13:20.883 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.899+0000 7faa3a5dc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:20.883 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.899+0000 7faa3a5dc640 1 -- 172.21.15.47:0/1633365370 >> v1:172.21.15.47:6790/0 conn(0x7faa3413b9d0 legacy=0x7faa3414aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:20.883 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.900+0000 7faa3a5dc640 1 -- 172.21.15.47:0/1633365370 shutdown_connections 2024-03-29T11:13:20.883 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.900+0000 7faa3a5dc640 1 -- 172.21.15.47:0/1633365370 >> 172.21.15.47:0/1633365370 conn(0x7faa340673e0 msgr2=0x7faa340b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:20.883 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.901+0000 7faa3a5dc640 1 -- 172.21.15.47:0/1633365370 shutdown_connections 2024-03-29T11:13:20.883 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.901+0000 7faa3a5dc640 1 -- 172.21.15.47:0/1633365370 wait complete. 2024-03-29T11:13:20.885 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.902+0000 7faa3a5dc640 10 monclient: build_initial_monmap 2024-03-29T11:13:20.885 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.902+0000 7faa3a5dc640 10 monclient: monmap: 2024-03-29T11:13:20.885 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:13:20.885 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:13:20.885 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:13:20.885 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:13:20.885 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:20.885 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:13:20.885 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:13:20.885 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:13:20.885 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:13:20.885 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:13:20.886 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.902+0000 7faa3a5dc640 1 librados: starting msgr at 2024-03-29T11:13:20.886 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.902+0000 7faa3a5dc640 1 librados: starting objecter 2024-03-29T11:13:20.886 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.903+0000 7faa3a5dc640 1 Processor -- start 2024-03-29T11:13:20.886 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.906+0000 7faa3a5dc640 1 -- start start 2024-03-29T11:13:20.886 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.906+0000 7faa3a5dc640 1 librados: setting wanted keys 2024-03-29T11:13:20.886 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.906+0000 7faa3a5dc640 1 librados: calling monclient init 2024-03-29T11:13:20.886 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.906+0000 7faa3a5dc640 10 monclient: init 2024-03-29T11:13:20.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.907+0000 7faa3a5dc640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:20.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.907+0000 7faa3a5dc640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:13:20.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.907+0000 7faa3a5dc640 10 monclient(hunting): picked mon.noname-a con 0x7faa340ae220 addr v1:172.21.15.47:6789/0 2024-03-29T11:13:20.889 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.911+0000 7faa3a5dc640 10 monclient(hunting): picked mon.noname-c con 0x7faa340b3f30 addr v1:172.21.15.47:6790/0 2024-03-29T11:13:20.889 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.911+0000 7faa38dd9640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7faa340b3f30 0x7faa341e3e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:52869/0 (socket says 172.21.15.47:52869) 2024-03-29T11:13:20.890 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.911+0000 7faa38dd9640 1 -- 172.21.15.47:0/2469281962 learned_addr learned my addr 172.21.15.47:0/2469281962 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:13:20.890 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.911+0000 7faa3a5dc640 10 monclient(hunting): picked mon.noname-b con 0x7faa3413b9d0 addr v1:172.21.15.121:6789/0 2024-03-29T11:13:20.890 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.911+0000 7faa3a5dc640 1 -- 172.21.15.47:0/2469281962 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa3406d8c0 con 0x7faa340ae220 2024-03-29T11:13:20.890 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.911+0000 7faa3a5dc640 1 -- 172.21.15.47:0/2469281962 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa3413be30 con 0x7faa3413b9d0 2024-03-29T11:13:20.890 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.911+0000 7faa3a5dc640 1 -- 172.21.15.47:0/2469281962 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa3413bfc0 con 0x7faa340b3f30 2024-03-29T11:13:20.890 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.911+0000 7faa3a5dc640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:20.890 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.911+0000 7faa3a5dc640 10 monclient(hunting): authenticate will time out at 844.099676s 2024-03-29T11:13:20.890 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.912+0000 7faa29ffb640 10 client.?.objecter ms_handle_connect 0x7faa340b3f30 2024-03-29T11:13:20.890 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.912+0000 7faa29ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:20.901 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.913+0000 7faa29ffb640 10 client.?.objecter ms_handle_connect 0x7faa340ae220 2024-03-29T11:13:20.901 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.913+0000 7faa29ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:20.901 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.913+0000 7faa29ffb640 1 -- 172.21.15.47:0/2469281962 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 999177887 0 0) 0x7faa300038c0 con 0x7faa340b3f30 2024-03-29T11:13:20.901 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.913+0000 7faa29ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:13:20.901 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.913+0000 7faa29ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:20.902 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.914+0000 7faa29ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:20.902 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.914+0000 7faa29ffb640 10 monclient(hunting): my global_id is 4256 2024-03-29T11:13:20.902 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.914+0000 7faa29ffb640 1 -- 172.21.15.47:0/2469281962 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa18004eb0 con 0x7faa340b3f30 2024-03-29T11:13:20.902 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.914+0000 7faa29ffb640 1 -- 172.21.15.47:0/2469281962 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3629390487 0 0) 0x7faa3406d8c0 con 0x7faa340ae220 2024-03-29T11:13:20.902 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.914+0000 7faa29ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:13:20.902 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.914+0000 7faa29ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:20.902 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.914+0000 7faa29ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:20.903 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.914+0000 7faa29ffb640 10 monclient(hunting): my global_id is 4254 2024-03-29T11:13:20.903 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.914+0000 7faa29ffb640 1 -- 172.21.15.47:0/2469281962 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa300038c0 con 0x7faa340ae220 2024-03-29T11:13:20.903 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.914+0000 7faa29ffb640 1 -- 172.21.15.47:0/2469281962 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3146451409 0 0) 0x7faa18004eb0 con 0x7faa340b3f30 2024-03-29T11:13:20.903 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.914+0000 7faa29ffb640 1 -- 172.21.15.47:0/2469281962 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa3406d8c0 con 0x7faa340b3f30 2024-03-29T11:13:20.903 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.915+0000 7faa29ffb640 1 -- 172.21.15.47:0/2469281962 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7faa30004070 con 0x7faa340b3f30 2024-03-29T11:13:20.903 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.915+0000 7faa29ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:20.903 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.915+0000 7faa29ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:13:20.903 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.915+0000 7faa29ffb640 10 monclient(hunting): dump: 2024-03-29T11:13:20.903 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:13:20.903 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:13:20.903 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:20.904 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:20.904 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:20.904 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:13:20.904 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:13:20.904 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:13:20.904 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:13:20.904 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:13:20.904 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.915+0000 7faa29ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:13:20.904 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.915+0000 7faa29ffb640 10 client.?.objecter ms_handle_connect 0x7faa3413b9d0 2024-03-29T11:13:20.904 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.915+0000 7faa29ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:20.904 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.915+0000 7faa29ffb640 1 -- 172.21.15.47:0/2469281962 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 21031000 0 0) 0x7faa3406d8c0 con 0x7faa340b3f30 2024-03-29T11:13:20.904 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.915+0000 7faa29ffb640 1 -- 172.21.15.47:0/2469281962 >> v1:172.21.15.121:6789/0 conn(0x7faa3413b9d0 legacy=0x7faa341ee4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:20.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.915+0000 7faa29ffb640 1 -- 172.21.15.47:0/2469281962 >> v1:172.21.15.47:6789/0 conn(0x7faa340ae220 legacy=0x7faa34070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:20.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.915+0000 7faa29ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:13:20.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.915+0000 7faa29ffb640 1 monclient: found mon.c 2024-03-29T11:13:20.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.915+0000 7faa29ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:20.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.915+0000 7faa29ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:13:20.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.915+0000 7faa29ffb640 1 -- 172.21.15.47:0/2469281962 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa34071490 con 0x7faa340b3f30 2024-03-29T11:13:20.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.915+0000 7faa29ffb640 10 monclient: _finish_auth 0 2024-03-29T11:13:20.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.915+0000 7faa29ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:13:20.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.915+0000 7faa29ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:20.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.915+0000 7faa3a5dc640 5 monclient: authenticate success, global_id 4256 2024-03-29T11:13:20.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.915+0000 7faa3a5dc640 10 monclient: _renew_subs 2024-03-29T11:13:20.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.915+0000 7faa3a5dc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:13:20.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.915+0000 7faa3a5dc640 1 -- 172.21.15.47:0/2469281962 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa34070070 con 0x7faa340b3f30 2024-03-29T11:13:20.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.916+0000 7faa3a5dc640 10 client.4256.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:13:20.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.916+0000 7faa3a5dc640 10 monclient: _renew_subs 2024-03-29T11:13:20.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.916+0000 7faa3a5dc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:13:20.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.916+0000 7faa3a5dc640 1 -- 172.21.15.47:0/2469281962 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faa341e4750 con 0x7faa340b3f30 2024-03-29T11:13:20.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.916+0000 7faa3a5dc640 1 librados: init done 2024-03-29T11:13:20.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.917+0000 7faa3a5dc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:20.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.917+0000 7faa3a5dc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:20.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.917+0000 7faa3a5dc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:13:20.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.917+0000 7faa3a5dc640 1 -- 172.21.15.47:0/2469281962 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa3404f730 con 0x7faa340b3f30 2024-03-29T11:13:20.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.918+0000 7faa29ffb640 1 -- 172.21.15.47:0/2469281962 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa300041d0 con 0x7faa340b3f30 2024-03-29T11:13:20.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.918+0000 7faa29ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:20.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.919+0000 7faa2b7fe640 4 set_mon_vals no callback set 2024-03-29T11:13:20.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.920+0000 7faa29ffb640 1 -- 172.21.15.47:0/2469281962 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7faa30001130 con 0x7faa340b3f30 2024-03-29T11:13:20.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.920+0000 7faa29ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:20.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.920+0000 7faa29ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:13:20.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.920+0000 7faa29ffb640 10 monclient: dump: 2024-03-29T11:13:20.907 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:13:20.908 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:13:20.908 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:20.908 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:20.908 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:20.908 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:13:20.909 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:13:20.909 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:13:20.909 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:13:20.909 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:13:20.909 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.920+0000 7faa29ffb640 1 -- 172.21.15.47:0/2469281962 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7faa30001310 con 0x7faa340b3f30 2024-03-29T11:13:20.909 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.921+0000 7faa29ffb640 1 -- 172.21.15.47:0/2469281962 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7faa30084b10 con 0x7faa340b3f30 2024-03-29T11:13:20.909 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.921+0000 7faa29ffb640 10 client.4256.objecter ms_dispatch 0x7faa34000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:13:20.909 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.921+0000 7faa29ffb640 3 client.4256.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:13:20.909 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.921+0000 7faa29ffb640 3 client.4256.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:13:20.909 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.921+0000 7faa29ffb640 20 client.4256.objecter dump_active .. 0 homeless 2024-03-29T11:13:20.909 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.921+0000 7faa395da640 10 monclient: get_auth_request con 0x7faa1806dd60 auth_method 0 2024-03-29T11:13:20.909 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.922+0000 7faa29ffb640 10 client.4256.objecter ms_handle_connect 0x7faa1806dd60 2024-03-29T11:13:20.910 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.929+0000 7faa29ffb640 1 -- 172.21.15.47:0/2469281962 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faa30050220 con 0x7faa340b3f30 2024-03-29T11:13:20.910 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.929+0000 7faa29ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:20.910 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:20.930+0000 7faa29ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:13:21.243 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:21.265+0000 7faa3a5dc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:21.243 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:21.265+0000 7faa3a5dc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:21.243 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:21.265+0000 7faa3a5dc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:13:21.243 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:21.265+0000 7faa3a5dc640 1 -- 172.21.15.47:0/2469281962 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faa340abb70 con 0x7faa340b3f30 2024-03-29T11:13:21.244 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:21.266+0000 7faa29ffb640 1 -- 172.21.15.47:0/2469281962 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7faa30052320 con 0x7faa340b3f30 2024-03-29T11:13:21.244 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:21.266+0000 7faa29ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:21.244 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:21.266+0000 7faa29ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:13:21.245 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:13:21.246 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:13:21.249 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:21.271+0000 7faa3a5dc640 10 librados: watch_flush enter 2024-03-29T11:13:21.250 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:21.271+0000 7faa3a5dc640 10 librados: watch_flush exit 2024-03-29T11:13:21.250 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:21.271+0000 7faa3a5dc640 20 client.4256.objecter shutdown clearing up homeless session... 2024-03-29T11:13:21.250 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:21.271+0000 7faa3a5dc640 10 client.4256.objecter successfully canceled tick 2024-03-29T11:13:21.250 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:21.271+0000 7faa3a5dc640 1 -- 172.21.15.47:0/2469281962 >> v1:172.21.15.121:6816/1964117370 conn(0x7faa1806dd60 legacy=0x7faa18070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:21.250 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:21.271+0000 7faa3a5dc640 10 monclient: shutdown 2024-03-29T11:13:21.250 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:21.271+0000 7faa3a5dc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:21.250 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:21.271+0000 7faa3a5dc640 1 -- 172.21.15.47:0/2469281962 >> v1:172.21.15.47:6790/0 conn(0x7faa340b3f30 legacy=0x7faa341e3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:21.250 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:21.272+0000 7faa3a5dc640 1 -- 172.21.15.47:0/2469281962 shutdown_connections 2024-03-29T11:13:21.250 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:21.272+0000 7faa3a5dc640 1 -- 172.21.15.47:0/2469281962 >> 172.21.15.47:0/2469281962 conn(0x7faa340673e0 msgr2=0x7faa34071f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:21.250 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:21.272+0000 7faa3a5dc640 1 -- 172.21.15.47:0/2469281962 shutdown_connections 2024-03-29T11:13:21.250 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:21.272+0000 7faa3a5dc640 1 -- 172.21.15.47:0/2469281962 wait complete. 2024-03-29T11:13:21.250 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:21.272+0000 7faa3a5dc640 1 librados: shutdown 2024-03-29T11:13:21.267 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:13:28.075 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:13:28.223 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.246+0000 7f416f49a640 10 monclient: get_monmap_and_config 2024-03-29T11:13:28.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.246+0000 7f416f49a640 10 monclient: build_initial_monmap 2024-03-29T11:13:28.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.246+0000 7f416f49a640 10 monclient: monmap: 2024-03-29T11:13:28.224 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:13:28.224 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:13:28.224 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:13:28.247205+0000 2024-03-29T11:13:28.224 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:13:28.247205+0000 2024-03-29T11:13:28.224 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:28.224 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:13:28.224 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:13:28.224 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:13:28.225 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:13:28.225 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:13:28.226 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.248+0000 7f416f49a640 1 Processor -- start 2024-03-29T11:13:28.227 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.249+0000 7f416f49a640 1 -- start start 2024-03-29T11:13:28.227 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.249+0000 7f416f49a640 10 monclient: init 2024-03-29T11:13:28.227 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.250+0000 7f416f49a640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:28.228 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.250+0000 7f416f49a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:13:28.228 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.250+0000 7f416f49a640 10 monclient(hunting): picked mon.noname-b con 0x7f41681422b0 addr v1:172.21.15.47:6790/0 2024-03-29T11:13:28.231 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.250+0000 7f416f49a640 10 monclient(hunting): picked mon.noname-c con 0x7f4168142690 addr v1:172.21.15.121:6789/0 2024-03-29T11:13:28.231 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.250+0000 7f416f49a640 10 monclient(hunting): picked mon.noname-a con 0x7f41680b17d0 addr v1:172.21.15.47:6789/0 2024-03-29T11:13:28.231 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.250+0000 7f416f49a640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4168142c80 con 0x7f41680b17d0 2024-03-29T11:13:28.231 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.250+0000 7f416f49a640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41680aae40 con 0x7f4168142690 2024-03-29T11:13:28.231 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.250+0000 7f416f49a640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f416813fbd0 con 0x7f41681422b0 2024-03-29T11:13:28.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.250+0000 7f416f49a640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:28.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.250+0000 7f416f49a640 10 monclient(hunting): authenticate will time out at 851.438138s 2024-03-29T11:13:28.233 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.254+0000 7f416da37640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f41680b17d0 0x7f41680b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:41255/0 (socket says 172.21.15.47:41255) 2024-03-29T11:13:28.233 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.254+0000 7f416da37640 1 -- 172.21.15.47:0/95986291 learned_addr learned my addr 172.21.15.47:0/95986291 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:13:28.233 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.254+0000 7f415ffff640 1 -- 172.21.15.47:0/95986291 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 246046618 0 0) 0x7f416813fbd0 con 0x7f41681422b0 2024-03-29T11:13:28.233 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.254+0000 7f415ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:13:28.233 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.254+0000 7f415ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:28.233 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.254+0000 7f415ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:28.233 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.254+0000 7f415ffff640 10 monclient(hunting): my global_id is 4259 2024-03-29T11:13:28.233 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.255+0000 7f415ffff640 1 -- 172.21.15.47:0/95986291 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4154003520 con 0x7f41681422b0 2024-03-29T11:13:28.233 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.255+0000 7f415ffff640 1 -- 172.21.15.47:0/95986291 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4230432314 0 0) 0x7f4154003520 con 0x7f41681422b0 2024-03-29T11:13:28.233 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.255+0000 7f415ffff640 1 -- 172.21.15.47:0/95986291 >> v1:172.21.15.121:6789/0 conn(0x7f4168142690 legacy=0x7f41680b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:13:28.233 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.255+0000 7f415ffff640 1 -- 172.21.15.47:0/95986291 >> v1:172.21.15.47:6789/0 conn(0x7f41680b17d0 legacy=0x7f41680b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:28.233 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.255+0000 7f415ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:13:28.234 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.255+0000 7f415ffff640 1 monclient: found mon.noname-b 2024-03-29T11:13:28.234 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.255+0000 7f415ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:28.234 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.255+0000 7f415ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-29T11:13:28.234 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.255+0000 7f415ffff640 1 -- 172.21.15.47:0/95986291 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4168151940 con 0x7f41681422b0 2024-03-29T11:13:28.234 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.255+0000 7f415ffff640 10 monclient: _finish_auth 0 2024-03-29T11:13:28.234 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.255+0000 7f415ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:13:28.234 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.255+0000 7f415ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:28.234 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.255+0000 7f415ffff640 1 -- 172.21.15.47:0/95986291 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f4160001a10 con 0x7f41681422b0 2024-03-29T11:13:28.234 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.255+0000 7f415ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:28.234 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.255+0000 7f415ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:13:28.234 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.255+0000 7f415ffff640 10 monclient: dump: 2024-03-29T11:13:28.234 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:13:28.234 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:13:28.235 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:28.235 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:28.235 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:28.235 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:13:28.235 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:13:28.235 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:13:28.235 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:13:28.235 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:13:28.235 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.255+0000 7f416f49a640 5 monclient: authenticate success, global_id 4259 2024-03-29T11:13:28.235 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.255+0000 7f416f49a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:13:28.236 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.255+0000 7f415ffff640 1 -- 172.21.15.47:0/95986291 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4160001b70 con 0x7f41681422b0 2024-03-29T11:13:28.236 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.255+0000 7f415ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:28.236 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.255+0000 7f415ffff640 1 -- 172.21.15.47:0/95986291 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f4160001fc0 con 0x7f41681422b0 2024-03-29T11:13:28.236 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.255+0000 7f415ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:28.236 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.255+0000 7f415ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:13:28.236 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.255+0000 7f415ffff640 10 monclient: dump: 2024-03-29T11:13:28.236 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:13:28.236 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:13:28.236 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:28.237 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:28.237 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:28.237 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:13:28.237 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:13:28.237 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:13:28.237 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:13:28.237 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:13:28.237 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.255+0000 7f416f49a640 10 monclient: get_monmap_and_config success 2024-03-29T11:13:28.237 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.255+0000 7f416f49a640 4 set_mon_vals no callback set 2024-03-29T11:13:28.237 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.256+0000 7f416f49a640 10 monclient: shutdown 2024-03-29T11:13:28.237 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.256+0000 7f416f49a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:28.237 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.256+0000 7f416f49a640 1 -- 172.21.15.47:0/95986291 >> v1:172.21.15.47:6790/0 conn(0x7f41681422b0 legacy=0x7f416813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:28.237 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.256+0000 7f416f49a640 1 -- 172.21.15.47:0/95986291 shutdown_connections 2024-03-29T11:13:28.237 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.256+0000 7f416f49a640 1 -- 172.21.15.47:0/95986291 >> 172.21.15.47:0/95986291 conn(0x7f4168066a00 msgr2=0x7f4168066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:28.240 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.262+0000 7f416f49a640 1 -- 172.21.15.47:0/95986291 shutdown_connections 2024-03-29T11:13:28.240 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.262+0000 7f416f49a640 1 -- 172.21.15.47:0/95986291 wait complete. 2024-03-29T11:13:28.242 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.264+0000 7f416f49a640 10 monclient: build_initial_monmap 2024-03-29T11:13:28.245 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.264+0000 7f416f49a640 10 monclient: monmap: 2024-03-29T11:13:28.245 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:13:28.245 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:13:28.245 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:13:28.245 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:13:28.245 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:28.246 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:13:28.246 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:13:28.246 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:13:28.246 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:13:28.246 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:13:28.246 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.264+0000 7f416f49a640 1 librados: starting msgr at 2024-03-29T11:13:28.246 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.264+0000 7f416f49a640 1 librados: starting objecter 2024-03-29T11:13:28.246 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.264+0000 7f416f49a640 1 Processor -- start 2024-03-29T11:13:28.246 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.264+0000 7f416f49a640 1 -- start start 2024-03-29T11:13:28.246 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.264+0000 7f416f49a640 1 librados: setting wanted keys 2024-03-29T11:13:28.246 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.264+0000 7f416f49a640 1 librados: calling monclient init 2024-03-29T11:13:28.246 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.264+0000 7f416f49a640 10 monclient: init 2024-03-29T11:13:28.246 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.265+0000 7f416f49a640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:28.246 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.265+0000 7f416f49a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:13:28.246 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.265+0000 7f416f49a640 10 monclient(hunting): picked mon.noname-b con 0x7f41680b17d0 addr v1:172.21.15.121:6789/0 2024-03-29T11:13:28.247 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.265+0000 7f416f49a640 10 monclient(hunting): picked mon.noname-c con 0x7f4168071750 addr v1:172.21.15.47:6790/0 2024-03-29T11:13:28.247 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.265+0000 7f416f49a640 10 monclient(hunting): picked mon.noname-a con 0x7f416806f140 addr v1:172.21.15.47:6789/0 2024-03-29T11:13:28.247 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.265+0000 7f416f49a640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41681e4230 con 0x7f416806f140 2024-03-29T11:13:28.247 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.265+0000 7f416f49a640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41680ad0f0 con 0x7f41680b17d0 2024-03-29T11:13:28.247 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.265+0000 7f416f49a640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41681e4dd0 con 0x7f4168071750 2024-03-29T11:13:28.247 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.265+0000 7f416f49a640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:28.247 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.265+0000 7f416f49a640 10 monclient(hunting): authenticate will time out at 851.453694s 2024-03-29T11:13:28.248 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.269+0000 7f416ca35640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f4168071750 0x7f41680ab7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:44485/0 (socket says 172.21.15.47:44485) 2024-03-29T11:13:28.248 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.269+0000 7f416ca35640 1 -- 172.21.15.47:0/1774144854 learned_addr learned my addr 172.21.15.47:0/1774144854 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:13:28.249 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.270+0000 7f415dffb640 10 client.?.objecter ms_handle_connect 0x7f4168071750 2024-03-29T11:13:28.249 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.270+0000 7f415dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:28.249 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.270+0000 7f415dffb640 1 -- 172.21.15.47:0/1774144854 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3998270941 0 0) 0x7f41681e4dd0 con 0x7f4168071750 2024-03-29T11:13:28.249 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.270+0000 7f415dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:13:28.249 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.270+0000 7f415dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:28.249 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.270+0000 7f415dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:28.249 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.270+0000 7f415dffb640 10 monclient(hunting): my global_id is 4262 2024-03-29T11:13:28.249 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.270+0000 7f415dffb640 1 -- 172.21.15.47:0/1774144854 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f414c004eb0 con 0x7f4168071750 2024-03-29T11:13:28.251 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.272+0000 7f415dffb640 10 client.?.objecter ms_handle_connect 0x7f416806f140 2024-03-29T11:13:28.251 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.272+0000 7f415dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:28.251 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.272+0000 7f415dffb640 10 client.?.objecter ms_handle_connect 0x7f41680b17d0 2024-03-29T11:13:28.251 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.272+0000 7f415dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:28.251 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.272+0000 7f415dffb640 1 -- 172.21.15.47:0/1774144854 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 730303772 0 0) 0x7f414c004eb0 con 0x7f4168071750 2024-03-29T11:13:28.251 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.272+0000 7f415dffb640 1 -- 172.21.15.47:0/1774144854 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f41681e4dd0 con 0x7f4168071750 2024-03-29T11:13:28.251 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.272+0000 7f415dffb640 1 -- 172.21.15.47:0/1774144854 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3440684770 0 0) 0x7f41681e4230 con 0x7f416806f140 2024-03-29T11:13:28.251 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.272+0000 7f415dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:13:28.252 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.272+0000 7f415dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:28.252 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.272+0000 7f415dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:28.252 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.272+0000 7f415dffb640 10 monclient(hunting): my global_id is 4257 2024-03-29T11:13:28.252 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.272+0000 7f415dffb640 1 -- 172.21.15.47:0/1774144854 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f414c004eb0 con 0x7f416806f140 2024-03-29T11:13:28.252 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.272+0000 7f415dffb640 1 -- 172.21.15.47:0/1774144854 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f4164001890 con 0x7f4168071750 2024-03-29T11:13:28.252 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.272+0000 7f415dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:28.252 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.272+0000 7f415dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:13:28.252 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.272+0000 7f415dffb640 10 monclient(hunting): dump: 2024-03-29T11:13:28.252 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:13:28.252 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:13:28.252 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:28.252 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:28.252 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:28.252 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:13:28.253 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:13:28.253 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:13:28.253 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:13:28.253 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:13:28.253 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.272+0000 7f415dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:13:28.253 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.272+0000 7f415dffb640 1 -- 172.21.15.47:0/1774144854 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1022200409 0 0) 0x7f41680ad0f0 con 0x7f41680b17d0 2024-03-29T11:13:28.253 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.272+0000 7f415dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:13:28.253 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.272+0000 7f415dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:28.253 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.272+0000 7f415dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:28.253 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.272+0000 7f415dffb640 10 monclient(hunting): my global_id is 4255 2024-03-29T11:13:28.253 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.272+0000 7f415dffb640 1 -- 172.21.15.47:0/1774144854 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41681e4230 con 0x7f41680b17d0 2024-03-29T11:13:28.253 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.272+0000 7f415dffb640 1 -- 172.21.15.47:0/1774144854 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1533252319 0 0) 0x7f41681e4dd0 con 0x7f4168071750 2024-03-29T11:13:28.253 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.272+0000 7f415dffb640 1 -- 172.21.15.47:0/1774144854 >> v1:172.21.15.121:6789/0 conn(0x7f41680b17d0 legacy=0x7f41681e46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:28.253 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.272+0000 7f415dffb640 1 -- 172.21.15.47:0/1774144854 >> v1:172.21.15.47:6789/0 conn(0x7f416806f140 legacy=0x7f416813be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:28.258 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.272+0000 7f415dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:13:28.258 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.272+0000 7f415dffb640 1 monclient: found mon.c 2024-03-29T11:13:28.258 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.272+0000 7f415dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:28.259 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.272+0000 7f415dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:13:28.259 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.272+0000 7f415dffb640 1 -- 172.21.15.47:0/1774144854 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41681e4f60 con 0x7f4168071750 2024-03-29T11:13:28.259 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.272+0000 7f415dffb640 10 monclient: _finish_auth 0 2024-03-29T11:13:28.259 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.272+0000 7f415dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:13:28.259 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.272+0000 7f415dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:28.259 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.272+0000 7f416f49a640 5 monclient: authenticate success, global_id 4262 2024-03-29T11:13:28.259 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.272+0000 7f416f49a640 10 monclient: _renew_subs 2024-03-29T11:13:28.259 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.272+0000 7f416f49a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:13:28.259 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.272+0000 7f416f49a640 1 -- 172.21.15.47:0/1774144854 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41680abec0 con 0x7f4168071750 2024-03-29T11:13:28.260 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.273+0000 7f416f49a640 10 client.4262.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:13:28.261 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.273+0000 7f416f49a640 10 monclient: _renew_subs 2024-03-29T11:13:28.261 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.273+0000 7f416f49a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:13:28.261 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.273+0000 7f416f49a640 1 -- 172.21.15.47:0/1774144854 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f416813c730 con 0x7f4168071750 2024-03-29T11:13:28.261 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.273+0000 7f416f49a640 1 librados: init done 2024-03-29T11:13:28.261 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.274+0000 7f41537fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:28.261 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.274+0000 7f41537fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:28.262 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.274+0000 7f41537fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:13:28.262 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.274+0000 7f41537fe640 1 -- 172.21.15.47:0/1774144854 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4148005080 con 0x7f4168071750 2024-03-29T11:13:28.262 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.275+0000 7f415dffb640 1 -- 172.21.15.47:0/1774144854 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f41640019f0 con 0x7f4168071750 2024-03-29T11:13:28.262 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.275+0000 7f415dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:28.262 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.275+0000 7f415f7fe640 4 set_mon_vals no callback set 2024-03-29T11:13:28.262 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.276+0000 7f415dffb640 1 -- 172.21.15.47:0/1774144854 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f4164003850 con 0x7f4168071750 2024-03-29T11:13:28.262 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.276+0000 7f415dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:28.262 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.276+0000 7f415dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:13:28.262 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.276+0000 7f415dffb640 10 monclient: dump: 2024-03-29T11:13:28.262 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:13:28.262 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:13:28.262 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:28.262 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:28.262 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:28.263 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:13:28.263 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:13:28.263 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:13:28.263 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:13:28.263 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:13:28.263 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.276+0000 7f415dffb640 1 -- 172.21.15.47:0/1774144854 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7f4164003a70 con 0x7f4168071750 2024-03-29T11:13:28.263 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.276+0000 7f415dffb640 1 -- 172.21.15.47:0/1774144854 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7f4164084a10 con 0x7f4168071750 2024-03-29T11:13:28.263 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.276+0000 7f415dffb640 10 client.4262.objecter ms_dispatch 0x7f4168000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:13:28.263 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.276+0000 7f415dffb640 3 client.4262.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:13:28.263 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.276+0000 7f415dffb640 3 client.4262.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:13:28.263 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.276+0000 7f415dffb640 20 client.4262.objecter dump_active .. 0 homeless 2024-03-29T11:13:28.263 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.277+0000 7f416d236640 10 monclient: get_auth_request con 0x7f414c06df20 auth_method 0 2024-03-29T11:13:28.263 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.280+0000 7f415dffb640 10 client.4262.objecter ms_handle_connect 0x7f414c06df20 2024-03-29T11:13:28.264 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.287+0000 7f415dffb640 1 -- 172.21.15.47:0/1774144854 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f41640501a0 con 0x7f4168071750 2024-03-29T11:13:28.264 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.287+0000 7f415dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:28.265 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.287+0000 7f415dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:13:28.582 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.603+0000 7f41537fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:28.582 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.604+0000 7f41537fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:28.582 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.604+0000 7f41537fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:13:28.582 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.604+0000 7f41537fe640 1 -- 172.21.15.47:0/1774144854 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4148005330 con 0x7f4168071750 2024-03-29T11:13:28.583 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.604+0000 7f415dffb640 1 -- 172.21.15.47:0/1774144854 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7f41640522a0 con 0x7f4168071750 2024-03-29T11:13:28.583 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.604+0000 7f415dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:28.583 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.604+0000 7f415dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:13:28.583 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:13:28.583 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:13:28.586 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.608+0000 7f41537fe640 10 librados: watch_flush enter 2024-03-29T11:13:28.586 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.608+0000 7f41537fe640 10 librados: watch_flush exit 2024-03-29T11:13:28.586 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.608+0000 7f41537fe640 20 client.4262.objecter shutdown clearing up homeless session... 2024-03-29T11:13:28.587 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.608+0000 7f41537fe640 10 client.4262.objecter successfully canceled tick 2024-03-29T11:13:28.587 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.609+0000 7f41537fe640 1 -- 172.21.15.47:0/1774144854 >> v1:172.21.15.121:6816/1964117370 conn(0x7f414c06df20 legacy=0x7f414c0703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:28.587 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.609+0000 7f41537fe640 10 monclient: shutdown 2024-03-29T11:13:28.587 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.609+0000 7f41537fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:28.587 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.609+0000 7f41537fe640 1 -- 172.21.15.47:0/1774144854 >> v1:172.21.15.47:6790/0 conn(0x7f4168071750 legacy=0x7f41680ab7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:28.588 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.610+0000 7f41537fe640 1 -- 172.21.15.47:0/1774144854 shutdown_connections 2024-03-29T11:13:28.588 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.610+0000 7f41537fe640 1 -- 172.21.15.47:0/1774144854 >> 172.21.15.47:0/1774144854 conn(0x7f4168066a00 msgr2=0x7f4168141870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:28.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.611+0000 7f41537fe640 1 -- 172.21.15.47:0/1774144854 shutdown_connections 2024-03-29T11:13:28.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.611+0000 7f41537fe640 1 -- 172.21.15.47:0/1774144854 wait complete. 2024-03-29T11:13:28.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:28.612+0000 7f41537fe640 1 librados: shutdown 2024-03-29T11:13:28.610 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:13:35.417 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:13:35.569 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.591+0000 7f2302f93640 10 monclient: get_monmap_and_config 2024-03-29T11:13:35.569 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.591+0000 7f2302f93640 10 monclient: build_initial_monmap 2024-03-29T11:13:35.569 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.591+0000 7f2302f93640 10 monclient: monmap: 2024-03-29T11:13:35.569 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:13:35.569 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:13:35.569 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:13:35.592020+0000 2024-03-29T11:13:35.569 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:13:35.592020+0000 2024-03-29T11:13:35.569 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:35.569 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:13:35.570 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:13:35.570 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:13:35.570 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:13:35.570 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:13:35.570 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.592+0000 7f2302f93640 1 Processor -- start 2024-03-29T11:13:35.570 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.592+0000 7f2302f93640 1 -- start start 2024-03-29T11:13:35.571 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.592+0000 7f2302f93640 10 monclient: init 2024-03-29T11:13:35.571 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.593+0000 7f2302f93640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:35.571 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.593+0000 7f2302f93640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:13:35.571 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.593+0000 7f2302f93640 10 monclient(hunting): picked mon.noname-a con 0x7f22fc0b0860 addr v1:172.21.15.47:6789/0 2024-03-29T11:13:35.571 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.593+0000 7f2302f93640 10 monclient(hunting): picked mon.noname-b con 0x7f22fc13ef30 addr v1:172.21.15.47:6790/0 2024-03-29T11:13:35.575 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.597+0000 7f2300d2f640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f22fc0b0860 0x7f22fc13e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:58769/0 (socket says 172.21.15.47:58769) 2024-03-29T11:13:35.575 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.597+0000 7f2302f93640 10 monclient(hunting): picked mon.noname-c con 0x7f22fc0af8e0 addr v1:172.21.15.121:6789/0 2024-03-29T11:13:35.575 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.597+0000 7f2300d2f640 1 -- 172.21.15.47:0/2737412434 learned_addr learned my addr 172.21.15.47:0/2737412434 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:13:35.575 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.597+0000 7f2302f93640 1 -- 172.21.15.47:0/2737412434 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22fc0b0c40 con 0x7f22fc0b0860 2024-03-29T11:13:35.575 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.597+0000 7f2302f93640 1 -- 172.21.15.47:0/2737412434 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22fc0aed10 con 0x7f22fc0af8e0 2024-03-29T11:13:35.575 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.597+0000 7f2302f93640 1 -- 172.21.15.47:0/2737412434 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22fc0afd50 con 0x7f22fc13ef30 2024-03-29T11:13:35.576 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.597+0000 7f2302f93640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:35.576 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.597+0000 7f2302f93640 10 monclient(hunting): authenticate will time out at 858.785867s 2024-03-29T11:13:35.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.598+0000 7f22fb7fe640 1 -- 172.21.15.47:0/2737412434 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4286408359 0 0) 0x7f22fc0afd50 con 0x7f22fc13ef30 2024-03-29T11:13:35.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.598+0000 7f22fb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:13:35.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.598+0000 7f22fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:35.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.598+0000 7f22fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:35.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.598+0000 7f22fb7fe640 10 monclient(hunting): my global_id is 4265 2024-03-29T11:13:35.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.598+0000 7f22fb7fe640 1 -- 172.21.15.47:0/2737412434 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22e8003520 con 0x7f22fc13ef30 2024-03-29T11:13:35.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.598+0000 7f22fb7fe640 1 -- 172.21.15.47:0/2737412434 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3428939605 0 0) 0x7f22fc0b0c40 con 0x7f22fc0b0860 2024-03-29T11:13:35.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.598+0000 7f22fb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:13:35.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.598+0000 7f22fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:35.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.598+0000 7f22fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:35.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.598+0000 7f22fb7fe640 10 monclient(hunting): my global_id is 4260 2024-03-29T11:13:35.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.598+0000 7f22fb7fe640 1 -- 172.21.15.47:0/2737412434 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22fc0afd50 con 0x7f22fc0b0860 2024-03-29T11:13:35.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.598+0000 7f22fb7fe640 1 -- 172.21.15.47:0/2737412434 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 982305896 0 0) 0x7f22fc0aed10 con 0x7f22fc0af8e0 2024-03-29T11:13:35.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.598+0000 7f22fb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:13:35.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.598+0000 7f22fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:35.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.598+0000 7f22fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:35.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.598+0000 7f22fb7fe640 10 monclient(hunting): my global_id is 4258 2024-03-29T11:13:35.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.598+0000 7f22fb7fe640 1 -- 172.21.15.47:0/2737412434 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22e8004bb0 con 0x7f22fc0af8e0 2024-03-29T11:13:35.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.599+0000 7f22fb7fe640 1 -- 172.21.15.47:0/2737412434 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3754800876 0 0) 0x7f22e8003520 con 0x7f22fc13ef30 2024-03-29T11:13:35.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.599+0000 7f22fb7fe640 1 -- 172.21.15.47:0/2737412434 >> v1:172.21.15.121:6789/0 conn(0x7f22fc0af8e0 legacy=0x7f22fc149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:35.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.599+0000 7f22fb7fe640 1 -- 172.21.15.47:0/2737412434 >> v1:172.21.15.47:6789/0 conn(0x7f22fc0b0860 legacy=0x7f22fc13e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:35.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.599+0000 7f22fb7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:13:35.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.599+0000 7f22fb7fe640 1 monclient: found mon.noname-b 2024-03-29T11:13:35.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.599+0000 7f22fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:35.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.599+0000 7f22fb7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-29T11:13:35.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.599+0000 7f22fb7fe640 1 -- 172.21.15.47:0/2737412434 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22fc0ab1b0 con 0x7f22fc13ef30 2024-03-29T11:13:35.580 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.599+0000 7f22fb7fe640 10 monclient: _finish_auth 0 2024-03-29T11:13:35.580 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.599+0000 7f22fb7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:13:35.580 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.599+0000 7f22fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:35.580 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.599+0000 7f22fb7fe640 1 -- 172.21.15.47:0/2737412434 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f22e4001690 con 0x7f22fc13ef30 2024-03-29T11:13:35.580 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.599+0000 7f22fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:35.580 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.599+0000 7f22fb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:13:35.580 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.599+0000 7f22fb7fe640 10 monclient: dump: 2024-03-29T11:13:35.580 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:13:35.580 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:13:35.580 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:35.580 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:35.580 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:35.580 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:13:35.580 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:13:35.580 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:13:35.581 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:13:35.581 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:13:35.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.599+0000 7f2302f93640 5 monclient: authenticate success, global_id 4265 2024-03-29T11:13:35.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.599+0000 7f2302f93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:13:35.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.600+0000 7f22fb7fe640 1 -- 172.21.15.47:0/2737412434 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22e40017f0 con 0x7f22fc13ef30 2024-03-29T11:13:35.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.600+0000 7f22fb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:35.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.600+0000 7f22fb7fe640 1 -- 172.21.15.47:0/2737412434 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f22e4001c00 con 0x7f22fc13ef30 2024-03-29T11:13:35.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.600+0000 7f22fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:35.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.600+0000 7f22fb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:13:35.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.600+0000 7f22fb7fe640 10 monclient: dump: 2024-03-29T11:13:35.581 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:13:35.581 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:13:35.581 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:35.581 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:35.581 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:35.582 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:13:35.582 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:13:35.582 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:13:35.582 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:13:35.582 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:13:35.582 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.600+0000 7f2302f93640 10 monclient: get_monmap_and_config success 2024-03-29T11:13:35.582 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.600+0000 7f2302f93640 4 set_mon_vals no callback set 2024-03-29T11:13:35.584 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.601+0000 7f2302f93640 10 monclient: shutdown 2024-03-29T11:13:35.585 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.601+0000 7f2302f93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:35.585 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.601+0000 7f2302f93640 1 -- 172.21.15.47:0/2737412434 >> v1:172.21.15.47:6790/0 conn(0x7f22fc13ef30 legacy=0x7f22fc0af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:35.585 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.601+0000 7f2302f93640 1 -- 172.21.15.47:0/2737412434 shutdown_connections 2024-03-29T11:13:35.585 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.601+0000 7f2302f93640 1 -- 172.21.15.47:0/2737412434 >> 172.21.15.47:0/2737412434 conn(0x7f22fc06b660 msgr2=0x7f22fc06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:35.585 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.601+0000 7f2302f93640 1 -- 172.21.15.47:0/2737412434 shutdown_connections 2024-03-29T11:13:35.585 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.602+0000 7f2302f93640 1 -- 172.21.15.47:0/2737412434 wait complete. 2024-03-29T11:13:35.585 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.603+0000 7f2302f93640 10 monclient: build_initial_monmap 2024-03-29T11:13:35.585 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.603+0000 7f2302f93640 10 monclient: monmap: 2024-03-29T11:13:35.585 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:13:35.585 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:13:35.585 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:13:35.585 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:13:35.585 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:35.585 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:13:35.585 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:13:35.586 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:13:35.586 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:13:35.586 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:13:35.586 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.604+0000 7f2302f93640 1 librados: starting msgr at 2024-03-29T11:13:35.586 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.604+0000 7f2302f93640 1 librados: starting objecter 2024-03-29T11:13:35.586 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.604+0000 7f2302f93640 1 Processor -- start 2024-03-29T11:13:35.587 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.604+0000 7f2302f93640 1 -- start start 2024-03-29T11:13:35.587 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.604+0000 7f2302f93640 1 librados: setting wanted keys 2024-03-29T11:13:35.587 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.604+0000 7f2302f93640 1 librados: calling monclient init 2024-03-29T11:13:35.587 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.604+0000 7f2302f93640 10 monclient: init 2024-03-29T11:13:35.587 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.605+0000 7f2302f93640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:35.587 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.605+0000 7f2302f93640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:13:35.587 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.606+0000 7f2302f93640 10 monclient(hunting): picked mon.noname-a con 0x7f22fc0af8e0 addr v1:172.21.15.47:6789/0 2024-03-29T11:13:35.587 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.607+0000 7f2302f93640 10 monclient(hunting): picked mon.noname-c con 0x7f22fc0b0860 addr v1:172.21.15.47:6790/0 2024-03-29T11:13:35.587 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.607+0000 7f2302f93640 10 monclient(hunting): picked mon.noname-b con 0x7f22fc13ef30 addr v1:172.21.15.121:6789/0 2024-03-29T11:13:35.588 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.607+0000 7f2302f93640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22fc1d3200 con 0x7f22fc0af8e0 2024-03-29T11:13:35.588 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.607+0000 7f2302f93640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22fc139ba0 con 0x7f22fc13ef30 2024-03-29T11:13:35.588 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.607+0000 7f2302f93640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22fc0b11d0 con 0x7f22fc0b0860 2024-03-29T11:13:35.588 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.607+0000 7f2302f93640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:35.588 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.607+0000 7f2302f93640 10 monclient(hunting): authenticate will time out at 858.795058s 2024-03-29T11:13:35.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.610+0000 7f2300d2f640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f22fc0af8e0 0x7f22fc0b2180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:58871/0 (socket says 172.21.15.47:58871) 2024-03-29T11:13:35.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.610+0000 7f2300d2f640 1 -- 172.21.15.47:0/2423258242 learned_addr learned my addr 172.21.15.47:0/2423258242 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:13:35.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.611+0000 7f22f97fa640 10 client.?.objecter ms_handle_connect 0x7f22fc0b0860 2024-03-29T11:13:35.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.611+0000 7f22f97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:35.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.611+0000 7f22f97fa640 1 -- 172.21.15.47:0/2423258242 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3366381218 0 0) 0x7f22fc0b11d0 con 0x7f22fc0b0860 2024-03-29T11:13:35.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.611+0000 7f22f97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:13:35.590 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.611+0000 7f22f97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:35.590 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.611+0000 7f22f97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:35.590 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.612+0000 7f22f97fa640 10 monclient(hunting): my global_id is 4268 2024-03-29T11:13:35.590 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.612+0000 7f22f97fa640 1 -- 172.21.15.47:0/2423258242 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22dc004eb0 con 0x7f22fc0b0860 2024-03-29T11:13:35.590 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.612+0000 7f22f97fa640 10 client.?.objecter ms_handle_connect 0x7f22fc0af8e0 2024-03-29T11:13:35.594 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.613+0000 7f22f97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:35.594 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.613+0000 7f22f97fa640 1 -- 172.21.15.47:0/2423258242 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3849782224 0 0) 0x7f22fc1d3200 con 0x7f22fc0af8e0 2024-03-29T11:13:35.594 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.613+0000 7f22f97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:13:35.594 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.613+0000 7f22f97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:35.594 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.613+0000 7f22f97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:35.595 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.613+0000 7f22f97fa640 10 monclient(hunting): my global_id is 4263 2024-03-29T11:13:35.595 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.613+0000 7f22f97fa640 1 -- 172.21.15.47:0/2423258242 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22fc0b11d0 con 0x7f22fc0af8e0 2024-03-29T11:13:35.595 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.613+0000 7f22f97fa640 1 -- 172.21.15.47:0/2423258242 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 116853624 0 0) 0x7f22fc0b11d0 con 0x7f22fc0af8e0 2024-03-29T11:13:35.595 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.613+0000 7f22f97fa640 1 -- 172.21.15.47:0/2423258242 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22fc1d3200 con 0x7f22fc0af8e0 2024-03-29T11:13:35.595 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.613+0000 7f22f97fa640 1 -- 172.21.15.47:0/2423258242 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f22ec0019f0 con 0x7f22fc0af8e0 2024-03-29T11:13:35.595 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.613+0000 7f22f97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:35.595 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.613+0000 7f22f97fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:13:35.595 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.613+0000 7f22f97fa640 10 monclient(hunting): dump: 2024-03-29T11:13:35.595 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:13:35.595 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:13:35.595 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:35.595 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:35.595 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:35.595 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:13:35.595 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:13:35.596 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:13:35.596 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:13:35.596 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:13:35.596 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.613+0000 7f22f97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:13:35.596 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.613+0000 7f22f97fa640 1 -- 172.21.15.47:0/2423258242 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2101047169 0 0) 0x7f22fc1d3200 con 0x7f22fc0af8e0 2024-03-29T11:13:35.596 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.613+0000 7f22f97fa640 1 -- 172.21.15.47:0/2423258242 >> v1:172.21.15.47:6790/0 conn(0x7f22fc0b0860 legacy=0x7f22fc1d3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:35.596 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.613+0000 7f22f97fa640 1 -- 172.21.15.47:0/2423258242 >> v1:172.21.15.121:6789/0 conn(0x7f22fc13ef30 legacy=0x7f22fc132710 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:13:35.596 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.613+0000 7f22f97fa640 10 monclient: _finish_hunting 0 2024-03-29T11:13:35.596 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.613+0000 7f22f97fa640 1 monclient: found mon.a 2024-03-29T11:13:35.596 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.613+0000 7f22f97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:35.596 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.613+0000 7f22f97fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:13:35.596 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.613+0000 7f22f97fa640 1 -- 172.21.15.47:0/2423258242 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22fc0b4220 con 0x7f22fc0af8e0 2024-03-29T11:13:35.596 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.613+0000 7f22f97fa640 10 monclient: _finish_auth 0 2024-03-29T11:13:35.596 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.613+0000 7f22f97fa640 10 monclient: _check_auth_tickets 2024-03-29T11:13:35.596 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.613+0000 7f22f97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:35.597 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.613+0000 7f2302f93640 5 monclient: authenticate success, global_id 4263 2024-03-29T11:13:35.597 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.613+0000 7f2302f93640 10 monclient: _renew_subs 2024-03-29T11:13:35.597 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.613+0000 7f2302f93640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:13:35.597 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.613+0000 7f2302f93640 1 -- 172.21.15.47:0/2423258242 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22fc0b2890 con 0x7f22fc0af8e0 2024-03-29T11:13:35.597 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.614+0000 7f2302f93640 10 client.4263.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:13:35.597 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.614+0000 7f2302f93640 10 monclient: _renew_subs 2024-03-29T11:13:35.597 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.614+0000 7f2302f93640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:13:35.597 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.614+0000 7f2302f93640 1 -- 172.21.15.47:0/2423258242 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f22fc1d3f80 con 0x7f22fc0af8e0 2024-03-29T11:13:35.597 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.614+0000 7f2302f93640 1 librados: init done 2024-03-29T11:13:35.597 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.615+0000 7f22f97fa640 1 -- 172.21.15.47:0/2423258242 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22ec0047e0 con 0x7f22fc0af8e0 2024-03-29T11:13:35.597 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.615+0000 7f22f97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:35.597 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.615+0000 7f22f97fa640 1 -- 172.21.15.47:0/2423258242 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f22ec004af0 con 0x7f22fc0af8e0 2024-03-29T11:13:35.597 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.615+0000 7f22f97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:35.597 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.615+0000 7f22f97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:13:35.598 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.615+0000 7f22f97fa640 10 monclient: dump: 2024-03-29T11:13:35.598 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:13:35.598 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:13:35.598 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:35.598 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:35.598 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:35.598 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:13:35.598 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:13:35.598 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:13:35.598 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:13:35.598 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:13:35.598 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.615+0000 7f2302f93640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:35.598 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.615+0000 7f2302f93640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:35.598 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.615+0000 7f2302f93640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:13:35.598 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.615+0000 7f2302f93640 1 -- 172.21.15.47:0/2423258242 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22fc0b2c30 con 0x7f22fc0af8e0 2024-03-29T11:13:35.600 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.616+0000 7f22faffd640 4 set_mon_vals no callback set 2024-03-29T11:13:35.600 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.616+0000 7f22f97fa640 1 -- 172.21.15.47:0/2423258242 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7f22ec01ab40 con 0x7f22fc0af8e0 2024-03-29T11:13:35.600 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.617+0000 7f22f97fa640 1 -- 172.21.15.47:0/2423258242 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7f22ec084ae0 con 0x7f22fc0af8e0 2024-03-29T11:13:35.600 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.617+0000 7f22f97fa640 10 client.4263.objecter ms_dispatch 0x7f22fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:13:35.600 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.617+0000 7f22f97fa640 3 client.4263.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:13:35.600 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.617+0000 7f22f97fa640 3 client.4263.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:13:35.600 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.617+0000 7f22fbfff640 10 monclient: get_auth_request con 0x7f22dc06dd60 auth_method 0 2024-03-29T11:13:35.600 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.617+0000 7f22f97fa640 20 client.4263.objecter dump_active .. 0 homeless 2024-03-29T11:13:35.600 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.620+0000 7f22f97fa640 10 client.4263.objecter ms_handle_connect 0x7f22dc06dd60 2024-03-29T11:13:35.603 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.625+0000 7f22f97fa640 1 -- 172.21.15.47:0/2423258242 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f22ec0501f0 con 0x7f22fc0af8e0 2024-03-29T11:13:35.603 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.625+0000 7f22f97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:35.603 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.625+0000 7f22f97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:13:35.940 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.962+0000 7f2302f93640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:35.940 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.962+0000 7f2302f93640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:35.940 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.962+0000 7f2302f93640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:13:35.940 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.962+0000 7f2302f93640 1 -- 172.21.15.47:0/2423258242 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f22fc075f10 con 0x7f22fc0af8e0 2024-03-29T11:13:35.944 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:13:35.944 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:13:35.944 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.966+0000 7f22f97fa640 1 -- 172.21.15.47:0/2423258242 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7f22ec0522f0 con 0x7f22fc0af8e0 2024-03-29T11:13:35.944 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.966+0000 7f22f97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:35.945 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.966+0000 7f22f97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:13:35.948 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.969+0000 7f2302f93640 10 librados: watch_flush enter 2024-03-29T11:13:35.948 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.969+0000 7f2302f93640 10 librados: watch_flush exit 2024-03-29T11:13:35.948 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.969+0000 7f2302f93640 20 client.4263.objecter shutdown clearing up homeless session... 2024-03-29T11:13:35.949 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.969+0000 7f2302f93640 10 client.4263.objecter successfully canceled tick 2024-03-29T11:13:35.949 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.970+0000 7f2302f93640 1 -- 172.21.15.47:0/2423258242 >> v1:172.21.15.121:6816/1964117370 conn(0x7f22dc06dd60 legacy=0x7f22dc070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:35.949 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.970+0000 7f2302f93640 10 monclient: shutdown 2024-03-29T11:13:35.949 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.970+0000 7f2302f93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:35.949 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.970+0000 7f2302f93640 1 -- 172.21.15.47:0/2423258242 >> v1:172.21.15.47:6789/0 conn(0x7f22fc0af8e0 legacy=0x7f22fc0b2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:35.949 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.971+0000 7f2302f93640 1 -- 172.21.15.47:0/2423258242 shutdown_connections 2024-03-29T11:13:35.949 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.971+0000 7f2302f93640 1 -- 172.21.15.47:0/2423258242 >> 172.21.15.47:0/2423258242 conn(0x7f22fc06b660 msgr2=0x7f22fc13e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:35.949 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.972+0000 7f2302f93640 1 -- 172.21.15.47:0/2423258242 shutdown_connections 2024-03-29T11:13:35.950 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.972+0000 7f2302f93640 1 -- 172.21.15.47:0/2423258242 wait complete. 2024-03-29T11:13:35.950 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:35.972+0000 7f2302f93640 1 librados: shutdown 2024-03-29T11:13:35.966 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:13:42.775 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:13:42.935 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.956+0000 7f4c74055640 10 monclient: get_monmap_and_config 2024-03-29T11:13:42.935 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.956+0000 7f4c74055640 10 monclient: build_initial_monmap 2024-03-29T11:13:42.935 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.956+0000 7f4c74055640 10 monclient: monmap: 2024-03-29T11:13:42.935 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:13:42.935 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:13:42.935 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:13:42.957644+0000 2024-03-29T11:13:42.935 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:13:42.957644+0000 2024-03-29T11:13:42.936 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:42.936 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:13:42.936 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:13:42.936 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:13:42.936 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:13:42.936 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:13:42.936 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.958+0000 7f4c74055640 1 Processor -- start 2024-03-29T11:13:42.936 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.958+0000 7f4c74055640 1 -- start start 2024-03-29T11:13:42.937 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.958+0000 7f4c74055640 10 monclient: init 2024-03-29T11:13:42.938 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.959+0000 7f4c74055640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:42.938 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.959+0000 7f4c74055640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:13:42.938 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.959+0000 7f4c74055640 10 monclient(hunting): picked mon.noname-c con 0x7f4c6c0b4100 addr v1:172.21.15.121:6789/0 2024-03-29T11:13:42.938 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.959+0000 7f4c74055640 10 monclient(hunting): picked mon.noname-a con 0x7f4c6c13ee90 addr v1:172.21.15.47:6789/0 2024-03-29T11:13:42.938 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.959+0000 7f4c74055640 10 monclient(hunting): picked mon.noname-b con 0x7f4c6c0af900 addr v1:172.21.15.47:6790/0 2024-03-29T11:13:42.938 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.959+0000 7f4c74055640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c6c0afd10 con 0x7f4c6c13ee90 2024-03-29T11:13:42.938 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.959+0000 7f4c74055640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c6c0af6f0 con 0x7f4c6c0b4100 2024-03-29T11:13:42.938 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.959+0000 7f4c74055640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c6c0aaf60 con 0x7f4c6c0af900 2024-03-29T11:13:42.938 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.959+0000 7f4c74055640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:42.938 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.959+0000 7f4c74055640 10 monclient(hunting): authenticate will time out at 866.147864s 2024-03-29T11:13:42.942 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.964+0000 7f4c715f0640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f4c6c13ee90 0x7f4c6c0adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:34753/0 (socket says 172.21.15.47:34753) 2024-03-29T11:13:42.942 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.964+0000 7f4c715f0640 1 -- 172.21.15.47:0/620697811 learned_addr learned my addr 172.21.15.47:0/620697811 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:13:42.942 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.965+0000 7f4c70def640 1 -- 172.21.15.47:0/620697811 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 687812479 0 0) 0x7f4c6c0afd10 con 0x7f4c6c13ee90 2024-03-29T11:13:42.942 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.965+0000 7f4c70def640 10 monclient(hunting): _negotiate 2024-03-29T11:13:42.942 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.965+0000 7f4c70def640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:42.942 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.965+0000 7f4c70def640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:42.943 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.965+0000 7f4c70def640 10 monclient(hunting): my global_id is 4266 2024-03-29T11:13:42.943 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.965+0000 7f4c70def640 1 -- 172.21.15.47:0/620697811 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c58003520 con 0x7f4c6c13ee90 2024-03-29T11:13:42.956 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.969+0000 7f4c70def640 1 -- 172.21.15.47:0/620697811 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3419889271 0 0) 0x7f4c58003520 con 0x7f4c6c13ee90 2024-03-29T11:13:42.956 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.969+0000 7f4c70def640 1 -- 172.21.15.47:0/620697811 >> v1:172.21.15.47:6790/0 conn(0x7f4c6c0af900 legacy=0x7f4c6c13b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:13:42.956 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.969+0000 7f4c70def640 1 -- 172.21.15.47:0/620697811 >> v1:172.21.15.121:6789/0 conn(0x7f4c6c0b4100 legacy=0x7f4c6c0a9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:13:42.956 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.969+0000 7f4c70def640 10 monclient: _finish_hunting 0 2024-03-29T11:13:42.956 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.969+0000 7f4c70def640 1 monclient: found mon.noname-a 2024-03-29T11:13:42.956 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.969+0000 7f4c70def640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:42.957 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.969+0000 7f4c70def640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-29T11:13:42.958 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.969+0000 7f4c70def640 1 -- 172.21.15.47:0/620697811 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c6c13e970 con 0x7f4c6c13ee90 2024-03-29T11:13:42.958 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.969+0000 7f4c70def640 10 monclient: _finish_auth 0 2024-03-29T11:13:42.958 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.969+0000 7f4c70def640 10 monclient: _check_auth_tickets 2024-03-29T11:13:42.958 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.970+0000 7f4c70def640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:42.958 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.970+0000 7f4c70def640 1 -- 172.21.15.47:0/620697811 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f4c64001a10 con 0x7f4c6c13ee90 2024-03-29T11:13:42.958 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.970+0000 7f4c70def640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:42.958 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.970+0000 7f4c70def640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:13:42.958 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.970+0000 7f4c70def640 10 monclient: dump: 2024-03-29T11:13:42.958 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:13:42.958 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:13:42.959 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:42.959 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:42.959 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:42.959 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:13:42.959 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:13:42.959 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:13:42.959 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:13:42.959 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:13:42.960 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.970+0000 7f4c74055640 5 monclient: authenticate success, global_id 4266 2024-03-29T11:13:42.960 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.970+0000 7f4c74055640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:13:42.960 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.972+0000 7f4c70def640 1 -- 172.21.15.47:0/620697811 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c64001ca0 con 0x7f4c6c13ee90 2024-03-29T11:13:42.960 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.972+0000 7f4c70def640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:42.960 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.972+0000 7f4c74055640 10 monclient: get_monmap_and_config success 2024-03-29T11:13:42.960 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.972+0000 7f4c74055640 4 set_mon_vals no callback set 2024-03-29T11:13:42.960 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.973+0000 7f4c74055640 10 monclient: shutdown 2024-03-29T11:13:42.960 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.973+0000 7f4c74055640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:42.961 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.973+0000 7f4c74055640 1 -- 172.21.15.47:0/620697811 >> v1:172.21.15.47:6789/0 conn(0x7f4c6c13ee90 legacy=0x7f4c6c0adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:42.961 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.973+0000 7f4c74055640 1 -- 172.21.15.47:0/620697811 shutdown_connections 2024-03-29T11:13:42.961 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.973+0000 7f4c74055640 1 -- 172.21.15.47:0/620697811 >> 172.21.15.47:0/620697811 conn(0x7f4c6c0b35e0 msgr2=0x7f4c6c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:42.961 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.974+0000 7f4c74055640 1 -- 172.21.15.47:0/620697811 shutdown_connections 2024-03-29T11:13:42.961 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.974+0000 7f4c74055640 1 -- 172.21.15.47:0/620697811 wait complete. 2024-03-29T11:13:42.961 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.975+0000 7f4c74055640 10 monclient: build_initial_monmap 2024-03-29T11:13:42.961 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.975+0000 7f4c74055640 10 monclient: monmap: 2024-03-29T11:13:42.961 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:13:42.961 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:13:42.962 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:13:42.962 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:13:42.962 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:42.962 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:13:42.962 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:13:42.962 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:13:42.962 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:13:42.962 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:13:42.962 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.976+0000 7f4c74055640 1 librados: starting msgr at 2024-03-29T11:13:42.963 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.976+0000 7f4c74055640 1 librados: starting objecter 2024-03-29T11:13:42.963 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.976+0000 7f4c74055640 1 Processor -- start 2024-03-29T11:13:42.963 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.976+0000 7f4c74055640 1 -- start start 2024-03-29T11:13:42.963 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.976+0000 7f4c74055640 1 librados: setting wanted keys 2024-03-29T11:13:42.963 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.976+0000 7f4c74055640 1 librados: calling monclient init 2024-03-29T11:13:42.963 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.976+0000 7f4c74055640 10 monclient: init 2024-03-29T11:13:42.963 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.977+0000 7f4c74055640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:42.963 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.977+0000 7f4c74055640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:13:42.963 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.977+0000 7f4c74055640 10 monclient(hunting): picked mon.noname-c con 0x7f4c6c0af900 addr v1:172.21.15.47:6790/0 2024-03-29T11:13:42.963 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.977+0000 7f4c74055640 10 monclient(hunting): picked mon.noname-a con 0x7f4c6c0b4100 addr v1:172.21.15.47:6789/0 2024-03-29T11:13:42.964 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.977+0000 7f4c74055640 10 monclient(hunting): picked mon.noname-b con 0x7f4c6c13ee90 addr v1:172.21.15.121:6789/0 2024-03-29T11:13:42.964 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.977+0000 7f4c74055640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c6c071820 con 0x7f4c6c0b4100 2024-03-29T11:13:42.964 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.977+0000 7f4c74055640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c6c12d0f0 con 0x7f4c6c13ee90 2024-03-29T11:13:42.964 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.977+0000 7f4c74055640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c6c071a90 con 0x7f4c6c0af900 2024-03-29T11:13:42.964 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.977+0000 7f4c74055640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:42.964 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.977+0000 7f4c74055640 10 monclient(hunting): authenticate will time out at 866.165605s 2024-03-29T11:13:42.970 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.983+0000 7f4c725f2640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f4c6c13ee90 0x7f4c6c12c9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.47:44943/0 (socket says 172.21.15.47:44943) 2024-03-29T11:13:42.971 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.983+0000 7f4c725f2640 1 -- 172.21.15.47:0/3577619594 learned_addr learned my addr 172.21.15.47:0/3577619594 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:13:42.971 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.983+0000 7f4c627fc640 10 client.?.objecter ms_handle_connect 0x7f4c6c13ee90 2024-03-29T11:13:42.971 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.983+0000 7f4c627fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:42.971 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.984+0000 7f4c627fc640 1 -- 172.21.15.47:0/3577619594 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1939669083 0 0) 0x7f4c6c12d0f0 con 0x7f4c6c13ee90 2024-03-29T11:13:42.971 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.984+0000 7f4c627fc640 10 monclient(hunting): _negotiate 2024-03-29T11:13:42.971 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.984+0000 7f4c627fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:42.971 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.984+0000 7f4c627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:42.971 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.984+0000 7f4c627fc640 10 monclient(hunting): my global_id is 4261 2024-03-29T11:13:42.972 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.984+0000 7f4c627fc640 1 -- 172.21.15.47:0/3577619594 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c50004eb0 con 0x7f4c6c13ee90 2024-03-29T11:13:42.972 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.984+0000 7f4c627fc640 10 client.?.objecter ms_handle_connect 0x7f4c6c0b4100 2024-03-29T11:13:42.972 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.984+0000 7f4c627fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:42.972 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.984+0000 7f4c627fc640 1 -- 172.21.15.47:0/3577619594 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 973376165 0 0) 0x7f4c50004eb0 con 0x7f4c6c13ee90 2024-03-29T11:13:42.972 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.984+0000 7f4c627fc640 1 -- 172.21.15.47:0/3577619594 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4c6c12d0f0 con 0x7f4c6c13ee90 2024-03-29T11:13:42.972 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.984+0000 7f4c627fc640 1 -- 172.21.15.47:0/3577619594 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f4c68004000 con 0x7f4c6c13ee90 2024-03-29T11:13:42.972 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.984+0000 7f4c627fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:42.972 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.984+0000 7f4c627fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:13:42.972 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.984+0000 7f4c627fc640 10 monclient(hunting): dump: 2024-03-29T11:13:42.972 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:13:42.973 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:13:42.973 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:42.973 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:42.973 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:42.973 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:13:42.973 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:13:42.973 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:13:42.973 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:13:42.973 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:13:42.974 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.984+0000 7f4c627fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:13:42.974 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.984+0000 7f4c627fc640 1 -- 172.21.15.47:0/3577619594 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 520535947 0 0) 0x7f4c6c071820 con 0x7f4c6c0b4100 2024-03-29T11:13:42.974 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.984+0000 7f4c627fc640 10 monclient(hunting): _negotiate 2024-03-29T11:13:42.974 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.984+0000 7f4c627fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:42.974 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.984+0000 7f4c627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:42.974 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.984+0000 7f4c627fc640 10 monclient(hunting): my global_id is 4269 2024-03-29T11:13:42.974 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.984+0000 7f4c627fc640 1 -- 172.21.15.47:0/3577619594 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c50004eb0 con 0x7f4c6c0b4100 2024-03-29T11:13:42.974 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.986+0000 7f4c627fc640 10 client.?.objecter ms_handle_connect 0x7f4c6c0af900 2024-03-29T11:13:42.974 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.987+0000 7f4c627fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:42.974 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.987+0000 7f4c627fc640 1 -- 172.21.15.47:0/3577619594 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1125260906 0 0) 0x7f4c6c12d0f0 con 0x7f4c6c13ee90 2024-03-29T11:13:42.975 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.987+0000 7f4c627fc640 1 -- 172.21.15.47:0/3577619594 >> v1:172.21.15.47:6790/0 conn(0x7f4c6c0af900 legacy=0x7f4c6c0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:42.975 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.987+0000 7f4c627fc640 1 -- 172.21.15.47:0/3577619594 >> v1:172.21.15.47:6789/0 conn(0x7f4c6c0b4100 legacy=0x7f4c6c12c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:42.975 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.987+0000 7f4c627fc640 10 monclient: _finish_hunting 0 2024-03-29T11:13:42.975 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.987+0000 7f4c627fc640 1 monclient: found mon.b 2024-03-29T11:13:42.975 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.987+0000 7f4c627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:42.975 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.987+0000 7f4c627fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:13:42.975 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.987+0000 7f4c627fc640 1 -- 172.21.15.47:0/3577619594 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c6c071d70 con 0x7f4c6c13ee90 2024-03-29T11:13:42.975 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.987+0000 7f4c627fc640 10 monclient: _finish_auth 0 2024-03-29T11:13:42.975 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.987+0000 7f4c627fc640 10 monclient: _check_auth_tickets 2024-03-29T11:13:42.975 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.987+0000 7f4c627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:42.975 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.987+0000 7f4c74055640 5 monclient: authenticate success, global_id 4261 2024-03-29T11:13:42.976 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.987+0000 7f4c74055640 10 monclient: _renew_subs 2024-03-29T11:13:42.976 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.987+0000 7f4c74055640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:13:42.976 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.987+0000 7f4c74055640 1 -- 172.21.15.47:0/3577619594 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c6c072050 con 0x7f4c6c13ee90 2024-03-29T11:13:42.976 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.987+0000 7f4c74055640 10 client.4261.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:13:42.976 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.987+0000 7f4c74055640 10 monclient: _renew_subs 2024-03-29T11:13:42.976 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.987+0000 7f4c74055640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:13:42.976 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.987+0000 7f4c74055640 1 -- 172.21.15.47:0/3577619594 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c6c072470 con 0x7f4c6c13ee90 2024-03-29T11:13:42.976 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.987+0000 7f4c74055640 1 librados: init done 2024-03-29T11:13:42.976 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.988+0000 7f4c74055640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:42.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.988+0000 7f4c74055640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:42.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.988+0000 7f4c74055640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:13:42.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.988+0000 7f4c74055640 1 -- 172.21.15.47:0/3577619594 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c6c14cde0 con 0x7f4c6c13ee90 2024-03-29T11:13:42.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.989+0000 7f4c627fc640 1 -- 172.21.15.47:0/3577619594 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c68000ed0 con 0x7f4c6c13ee90 2024-03-29T11:13:42.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.990+0000 7f4c627fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:42.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.990+0000 7f4c63fff640 4 set_mon_vals no callback set 2024-03-29T11:13:42.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.991+0000 7f4c627fc640 1 -- 172.21.15.47:0/3577619594 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f4c68004160 con 0x7f4c6c13ee90 2024-03-29T11:13:42.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.991+0000 7f4c627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:42.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.991+0000 7f4c627fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:13:42.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.991+0000 7f4c627fc640 10 monclient: dump: 2024-03-29T11:13:42.977 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:13:42.978 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:13:42.978 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:42.978 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:42.978 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:42.978 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:13:42.978 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:13:42.978 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:13:42.978 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:13:42.978 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:13:42.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.991+0000 7f4c627fc640 1 -- 172.21.15.47:0/3577619594 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7f4c68001380 con 0x7f4c6c13ee90 2024-03-29T11:13:42.979 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.992+0000 7f4c627fc640 1 -- 172.21.15.47:0/3577619594 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7f4c68054860 con 0x7f4c6c13ee90 2024-03-29T11:13:42.979 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.992+0000 7f4c627fc640 10 client.4261.objecter ms_dispatch 0x7f4c6c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:13:42.979 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.992+0000 7f4c627fc640 3 client.4261.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:13:42.979 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.992+0000 7f4c627fc640 3 client.4261.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:13:42.979 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.992+0000 7f4c627fc640 20 client.4261.objecter dump_active .. 0 homeless 2024-03-29T11:13:42.979 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.992+0000 7f4c71df1640 10 monclient: get_auth_request con 0x7f4c5006ddb0 auth_method 0 2024-03-29T11:13:42.980 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.993+0000 7f4c627fc640 10 client.4261.objecter ms_handle_connect 0x7f4c5006ddb0 2024-03-29T11:13:42.980 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.998+0000 7f4c627fc640 1 -- 172.21.15.47:0/3577619594 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4c68050370 con 0x7f4c6c13ee90 2024-03-29T11:13:42.980 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.998+0000 7f4c627fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:42.980 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:42.998+0000 7f4c627fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:13:43.315 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:43.337+0000 7f4c74055640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:43.315 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:43.337+0000 7f4c74055640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:43.315 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:43.337+0000 7f4c74055640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:13:43.315 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:43.337+0000 7f4c74055640 1 -- 172.21.15.47:0/3577619594 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4c6c075f10 con 0x7f4c6c13ee90 2024-03-29T11:13:43.317 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:43.339+0000 7f4c627fc640 1 -- 172.21.15.47:0/3577619594 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7f4c680522a0 con 0x7f4c6c13ee90 2024-03-29T11:13:43.317 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:43.339+0000 7f4c627fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:43.317 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:43.339+0000 7f4c627fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:13:43.317 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:13:43.317 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:13:43.320 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:43.342+0000 7f4c74055640 10 librados: watch_flush enter 2024-03-29T11:13:43.321 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:43.342+0000 7f4c74055640 10 librados: watch_flush exit 2024-03-29T11:13:43.321 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:43.342+0000 7f4c74055640 20 client.4261.objecter shutdown clearing up homeless session... 2024-03-29T11:13:43.321 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:43.342+0000 7f4c74055640 10 client.4261.objecter successfully canceled tick 2024-03-29T11:13:43.321 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:43.344+0000 7f4c74055640 1 -- 172.21.15.47:0/3577619594 >> v1:172.21.15.121:6816/1964117370 conn(0x7f4c5006ddb0 legacy=0x7f4c50070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:43.322 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:43.344+0000 7f4c74055640 10 monclient: shutdown 2024-03-29T11:13:43.322 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:43.344+0000 7f4c74055640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:43.322 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:43.344+0000 7f4c74055640 1 -- 172.21.15.47:0/3577619594 >> v1:172.21.15.121:6789/0 conn(0x7f4c6c13ee90 legacy=0x7f4c6c12c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:43.322 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:43.344+0000 7f4c74055640 1 -- 172.21.15.47:0/3577619594 shutdown_connections 2024-03-29T11:13:43.322 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:43.344+0000 7f4c74055640 1 -- 172.21.15.47:0/3577619594 >> 172.21.15.47:0/3577619594 conn(0x7f4c6c0b35e0 msgr2=0x7f4c6c13ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:43.326 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:43.348+0000 7f4c74055640 1 -- 172.21.15.47:0/3577619594 shutdown_connections 2024-03-29T11:13:43.326 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:43.348+0000 7f4c74055640 1 -- 172.21.15.47:0/3577619594 wait complete. 2024-03-29T11:13:43.326 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:43.348+0000 7f4c74055640 1 librados: shutdown 2024-03-29T11:13:43.349 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:13:50.158 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:13:50.314 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.336+0000 7fee25638640 10 monclient: get_monmap_and_config 2024-03-29T11:13:50.315 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.336+0000 7fee25638640 10 monclient: build_initial_monmap 2024-03-29T11:13:50.315 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.336+0000 7fee25638640 10 monclient: monmap: 2024-03-29T11:13:50.315 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:13:50.315 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:13:50.315 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:13:50.337763+0000 2024-03-29T11:13:50.315 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:13:50.337763+0000 2024-03-29T11:13:50.316 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:50.316 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:13:50.316 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:13:50.316 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:13:50.316 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:13:50.316 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:13:50.318 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.338+0000 7fee25638640 1 Processor -- start 2024-03-29T11:13:50.320 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.341+0000 7fee25638640 1 -- start start 2024-03-29T11:13:50.320 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.341+0000 7fee25638640 10 monclient: init 2024-03-29T11:13:50.320 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.342+0000 7fee25638640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:50.320 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.342+0000 7fee25638640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:13:50.320 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.342+0000 7fee25638640 10 monclient(hunting): picked mon.noname-c con 0x7fee200b0770 addr v1:172.21.15.121:6789/0 2024-03-29T11:13:50.321 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.342+0000 7fee25638640 10 monclient(hunting): picked mon.noname-b con 0x7fee2014a700 addr v1:172.21.15.47:6790/0 2024-03-29T11:13:50.321 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.342+0000 7fee25638640 10 monclient(hunting): picked mon.noname-a con 0x7fee200ac5a0 addr v1:172.21.15.47:6789/0 2024-03-29T11:13:50.321 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.342+0000 7fee25638640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee200aba70 con 0x7fee200ac5a0 2024-03-29T11:13:50.321 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.342+0000 7fee25638640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee200aecb0 con 0x7fee200b0770 2024-03-29T11:13:50.322 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.343+0000 7fee25638640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee200af150 con 0x7fee2014a700 2024-03-29T11:13:50.322 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.343+0000 7fee25638640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:50.322 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.343+0000 7fee25638640 10 monclient(hunting): authenticate will time out at 873.531050s 2024-03-29T11:13:50.324 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.346+0000 7fee1f7fe640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fee200ac5a0 0x7fee20151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:48533/0 (socket says 172.21.15.47:48533) 2024-03-29T11:13:50.324 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.346+0000 7fee1f7fe640 1 -- 172.21.15.47:0/1758253286 learned_addr learned my addr 172.21.15.47:0/1758253286 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:13:50.330 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.346+0000 7fee1dffb640 1 -- 172.21.15.47:0/1758253286 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3506493009 0 0) 0x7fee200aba70 con 0x7fee200ac5a0 2024-03-29T11:13:50.330 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.346+0000 7fee1dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:13:50.331 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.346+0000 7fee1dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:50.331 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.346+0000 7fee1dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:50.331 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.346+0000 7fee1dffb640 10 monclient(hunting): my global_id is 4272 2024-03-29T11:13:50.331 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.346+0000 7fee1dffb640 1 -- 172.21.15.47:0/1758253286 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee08003520 con 0x7fee200ac5a0 2024-03-29T11:13:50.331 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.347+0000 7fee1dffb640 1 -- 172.21.15.47:0/1758253286 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1908173371 0 0) 0x7fee08003520 con 0x7fee200ac5a0 2024-03-29T11:13:50.331 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.347+0000 7fee1dffb640 1 -- 172.21.15.47:0/1758253286 >> v1:172.21.15.47:6790/0 conn(0x7fee2014a700 legacy=0x7fee200abe90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:13:50.331 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.347+0000 7fee1dffb640 1 -- 172.21.15.47:0/1758253286 >> v1:172.21.15.121:6789/0 conn(0x7fee200b0770 legacy=0x7fee20149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:13:50.331 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.347+0000 7fee1dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:13:50.331 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.347+0000 7fee1dffb640 1 monclient: found mon.noname-a 2024-03-29T11:13:50.332 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.347+0000 7fee1dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:50.332 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.347+0000 7fee1dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-29T11:13:50.332 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.347+0000 7fee1dffb640 1 -- 172.21.15.47:0/1758253286 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee200ac9d0 con 0x7fee200ac5a0 2024-03-29T11:13:50.332 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.347+0000 7fee1dffb640 10 monclient: _finish_auth 0 2024-03-29T11:13:50.332 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.347+0000 7fee1dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:13:50.332 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.347+0000 7fee1dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:50.332 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.347+0000 7fee1dffb640 1 -- 172.21.15.47:0/1758253286 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fee10001690 con 0x7fee200ac5a0 2024-03-29T11:13:50.332 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.347+0000 7fee1dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:50.332 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.347+0000 7fee1dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:13:50.332 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.347+0000 7fee1dffb640 10 monclient: dump: 2024-03-29T11:13:50.332 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:13:50.333 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:13:50.333 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:50.333 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:50.333 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:50.333 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:13:50.333 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:13:50.333 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:13:50.333 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:13:50.333 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:13:50.333 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.347+0000 7fee25638640 5 monclient: authenticate success, global_id 4272 2024-03-29T11:13:50.333 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.347+0000 7fee25638640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:13:50.333 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.347+0000 7fee1dffb640 1 -- 172.21.15.47:0/1758253286 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee10001860 con 0x7fee200ac5a0 2024-03-29T11:13:50.333 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.347+0000 7fee1dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:50.333 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.347+0000 7fee1dffb640 1 -- 172.21.15.47:0/1758253286 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fee10001cb0 con 0x7fee200ac5a0 2024-03-29T11:13:50.333 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.347+0000 7fee1dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:50.334 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.347+0000 7fee1dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:13:50.334 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.347+0000 7fee1dffb640 10 monclient: dump: 2024-03-29T11:13:50.334 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:13:50.334 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:13:50.334 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:50.334 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:50.334 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:50.334 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:13:50.334 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:13:50.334 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:13:50.334 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:13:50.334 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:13:50.334 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.348+0000 7fee25638640 10 monclient: get_monmap_and_config success 2024-03-29T11:13:50.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.348+0000 7fee25638640 4 set_mon_vals no callback set 2024-03-29T11:13:50.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.349+0000 7fee25638640 10 monclient: shutdown 2024-03-29T11:13:50.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.349+0000 7fee25638640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:50.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.349+0000 7fee25638640 1 -- 172.21.15.47:0/1758253286 >> v1:172.21.15.47:6789/0 conn(0x7fee200ac5a0 legacy=0x7fee20151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:50.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.349+0000 7fee25638640 1 -- 172.21.15.47:0/1758253286 shutdown_connections 2024-03-29T11:13:50.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.349+0000 7fee25638640 1 -- 172.21.15.47:0/1758253286 >> 172.21.15.47:0/1758253286 conn(0x7fee2006b660 msgr2=0x7fee2006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:50.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.349+0000 7fee25638640 1 -- 172.21.15.47:0/1758253286 shutdown_connections 2024-03-29T11:13:50.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.349+0000 7fee25638640 1 -- 172.21.15.47:0/1758253286 wait complete. 2024-03-29T11:13:50.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.350+0000 7fee25638640 10 monclient: build_initial_monmap 2024-03-29T11:13:50.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.350+0000 7fee25638640 10 monclient: monmap: 2024-03-29T11:13:50.335 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:13:50.335 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:13:50.335 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:13:50.335 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:13:50.335 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:50.336 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:13:50.336 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:13:50.336 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:13:50.336 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:13:50.336 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:13:50.336 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.350+0000 7fee25638640 1 librados: starting msgr at 2024-03-29T11:13:50.336 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.350+0000 7fee25638640 1 librados: starting objecter 2024-03-29T11:13:50.336 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.351+0000 7fee25638640 1 Processor -- start 2024-03-29T11:13:50.336 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.351+0000 7fee25638640 1 -- start start 2024-03-29T11:13:50.336 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.351+0000 7fee25638640 1 librados: setting wanted keys 2024-03-29T11:13:50.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.351+0000 7fee25638640 1 librados: calling monclient init 2024-03-29T11:13:50.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.351+0000 7fee25638640 10 monclient: init 2024-03-29T11:13:50.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.351+0000 7fee25638640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:50.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.351+0000 7fee25638640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:13:50.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.351+0000 7fee25638640 10 monclient(hunting): picked mon.noname-c con 0x7fee200ac5a0 addr v1:172.21.15.47:6790/0 2024-03-29T11:13:50.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.351+0000 7fee25638640 10 monclient(hunting): picked mon.noname-b con 0x7fee200b0770 addr v1:172.21.15.121:6789/0 2024-03-29T11:13:50.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.351+0000 7fee25638640 10 monclient(hunting): picked mon.noname-a con 0x7fee2014a700 addr v1:172.21.15.47:6789/0 2024-03-29T11:13:50.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.351+0000 7fee25638640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee20129560 con 0x7fee2014a700 2024-03-29T11:13:50.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.351+0000 7fee25638640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee2012a300 con 0x7fee200b0770 2024-03-29T11:13:50.338 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.351+0000 7fee25638640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee20147c00 con 0x7fee200ac5a0 2024-03-29T11:13:50.338 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.351+0000 7fee25638640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:50.338 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.352+0000 7fee25638640 10 monclient(hunting): authenticate will time out at 873.539934s 2024-03-29T11:13:50.343 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.355+0000 7fee1e7fc640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fee200b0770 0x7fee20146ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.47:48971/0 (socket says 172.21.15.47:48971) 2024-03-29T11:13:50.343 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.355+0000 7fee1e7fc640 1 -- 172.21.15.47:0/2994338131 learned_addr learned my addr 172.21.15.47:0/2994338131 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:13:50.343 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.356+0000 7fee077fe640 10 client.?.objecter ms_handle_connect 0x7fee200b0770 2024-03-29T11:13:50.343 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.356+0000 7fee077fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:50.343 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.356+0000 7fee077fe640 1 -- 172.21.15.47:0/2994338131 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1283720443 0 0) 0x7fee2012a300 con 0x7fee200b0770 2024-03-29T11:13:50.343 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.356+0000 7fee077fe640 10 monclient(hunting): _negotiate 2024-03-29T11:13:50.344 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.356+0000 7fee077fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:50.344 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.356+0000 7fee077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:50.344 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.356+0000 7fee077fe640 10 monclient(hunting): my global_id is 4264 2024-03-29T11:13:50.344 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.356+0000 7fee077fe640 1 -- 172.21.15.47:0/2994338131 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee0c004eb0 con 0x7fee200b0770 2024-03-29T11:13:50.344 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.356+0000 7fee077fe640 10 client.?.objecter ms_handle_connect 0x7fee200ac5a0 2024-03-29T11:13:50.344 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.356+0000 7fee077fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:50.344 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.357+0000 7fee077fe640 1 -- 172.21.15.47:0/2994338131 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 408768356 0 0) 0x7fee0c004eb0 con 0x7fee200b0770 2024-03-29T11:13:50.344 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.357+0000 7fee077fe640 1 -- 172.21.15.47:0/2994338131 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee0c005300 con 0x7fee200b0770 2024-03-29T11:13:50.344 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.357+0000 7fee077fe640 1 -- 172.21.15.47:0/2994338131 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fee14001790 con 0x7fee200b0770 2024-03-29T11:13:50.344 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.357+0000 7fee077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:50.344 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.357+0000 7fee077fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:13:50.344 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.357+0000 7fee077fe640 10 monclient(hunting): dump: 2024-03-29T11:13:50.344 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:13:50.345 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:13:50.345 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:50.345 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:50.345 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:50.345 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:13:50.345 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:13:50.345 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:13:50.345 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:13:50.345 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:13:50.345 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.357+0000 7fee077fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:13:50.345 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.357+0000 7fee077fe640 1 -- 172.21.15.47:0/2994338131 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3615926252 0 0) 0x7fee20147c00 con 0x7fee200ac5a0 2024-03-29T11:13:50.345 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.357+0000 7fee077fe640 10 monclient(hunting): _negotiate 2024-03-29T11:13:50.345 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.357+0000 7fee077fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:50.346 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.357+0000 7fee077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:50.346 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.357+0000 7fee077fe640 10 monclient(hunting): my global_id is 4274 2024-03-29T11:13:50.346 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.357+0000 7fee077fe640 1 -- 172.21.15.47:0/2994338131 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee0c005850 con 0x7fee200ac5a0 2024-03-29T11:13:50.346 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.357+0000 7fee077fe640 1 -- 172.21.15.47:0/2994338131 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1309637211 0 0) 0x7fee0c005300 con 0x7fee200b0770 2024-03-29T11:13:50.346 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.357+0000 7fee077fe640 1 -- 172.21.15.47:0/2994338131 >> v1:172.21.15.47:6790/0 conn(0x7fee200ac5a0 legacy=0x7fee20128180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:50.346 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.357+0000 7fee077fe640 1 -- 172.21.15.47:0/2994338131 >> v1:172.21.15.47:6789/0 conn(0x7fee2014a700 legacy=0x7fee201db960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:13:50.346 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.358+0000 7fee077fe640 10 monclient: _finish_hunting 0 2024-03-29T11:13:50.346 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.358+0000 7fee077fe640 1 monclient: found mon.b 2024-03-29T11:13:50.346 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.358+0000 7fee077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:50.346 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.358+0000 7fee077fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:13:50.346 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.358+0000 7fee077fe640 1 -- 172.21.15.47:0/2994338131 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee201290a0 con 0x7fee200b0770 2024-03-29T11:13:50.346 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.358+0000 7fee077fe640 10 monclient: _finish_auth 0 2024-03-29T11:13:50.346 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.358+0000 7fee077fe640 10 monclient: _check_auth_tickets 2024-03-29T11:13:50.347 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.358+0000 7fee077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:50.347 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.359+0000 7fee077fe640 1 -- 172.21.15.47:0/2994338131 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee14001d20 con 0x7fee200b0770 2024-03-29T11:13:50.347 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.359+0000 7fee077fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:50.347 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.359+0000 7fee1cff9640 4 set_mon_vals no callback set 2024-03-29T11:13:50.347 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.360+0000 7fee077fe640 1 -- 172.21.15.47:0/2994338131 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fee14001790 con 0x7fee200b0770 2024-03-29T11:13:50.347 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.360+0000 7fee077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:50.347 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.360+0000 7fee077fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:13:50.347 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.360+0000 7fee077fe640 10 monclient: dump: 2024-03-29T11:13:50.347 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:13:50.347 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:13:50.347 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:50.347 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:50.347 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:50.347 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:13:50.348 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:13:50.348 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:13:50.348 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:13:50.348 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:13:50.348 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.362+0000 7fee25638640 5 monclient: authenticate success, global_id 4264 2024-03-29T11:13:50.348 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.362+0000 7fee25638640 10 monclient: _renew_subs 2024-03-29T11:13:50.348 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.362+0000 7fee25638640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:13:50.348 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.362+0000 7fee25638640 1 -- 172.21.15.47:0/2994338131 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee20147600 con 0x7fee200b0770 2024-03-29T11:13:50.348 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.362+0000 7fee25638640 10 client.4264.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:13:50.348 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.362+0000 7fee25638640 10 monclient: _renew_subs 2024-03-29T11:13:50.348 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.362+0000 7fee25638640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:13:50.348 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.362+0000 7fee25638640 1 -- 172.21.15.47:0/2994338131 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fee201dc070 con 0x7fee200b0770 2024-03-29T11:13:50.348 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.362+0000 7fee25638640 1 librados: init done 2024-03-29T11:13:50.348 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.363+0000 7fee25638640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:50.348 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.363+0000 7fee25638640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:50.349 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.363+0000 7fee25638640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:13:50.349 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.363+0000 7fee25638640 1 -- 172.21.15.47:0/2994338131 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee20075f10 con 0x7fee200b0770 2024-03-29T11:13:50.350 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.366+0000 7fee077fe640 1 -- 172.21.15.47:0/2994338131 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7fee1401a7d0 con 0x7fee200b0770 2024-03-29T11:13:50.350 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.367+0000 7fee077fe640 1 -- 172.21.15.47:0/2994338131 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7fee140542b0 con 0x7fee200b0770 2024-03-29T11:13:50.350 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.367+0000 7fee077fe640 10 client.4264.objecter ms_dispatch 0x7fee20000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:13:50.350 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.367+0000 7fee077fe640 3 client.4264.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:13:50.350 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.367+0000 7fee077fe640 3 client.4264.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:13:50.350 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.367+0000 7fee077fe640 20 client.4264.objecter dump_active .. 0 homeless 2024-03-29T11:13:50.350 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.370+0000 7fee1effd640 10 monclient: get_auth_request con 0x7fee0c06e500 auth_method 0 2024-03-29T11:13:50.350 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.371+0000 7fee077fe640 10 client.4264.objecter ms_handle_connect 0x7fee0c06e500 2024-03-29T11:13:50.351 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.373+0000 7fee077fe640 1 -- 172.21.15.47:0/2994338131 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fee14050550 con 0x7fee200b0770 2024-03-29T11:13:50.351 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.373+0000 7fee077fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:50.351 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.373+0000 7fee077fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:13:50.685 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.707+0000 7fee25638640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:50.685 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.707+0000 7fee25638640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:50.685 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.707+0000 7fee25638640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:13:50.685 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.707+0000 7fee25638640 1 -- 172.21.15.47:0/2994338131 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee200ac110 con 0x7fee200b0770 2024-03-29T11:13:50.687 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.709+0000 7fee077fe640 1 -- 172.21.15.47:0/2994338131 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7fee14050550 con 0x7fee200b0770 2024-03-29T11:13:50.687 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.710+0000 7fee077fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:50.687 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.710+0000 7fee077fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:13:50.688 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:13:50.688 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:13:50.691 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.713+0000 7fee25638640 10 librados: watch_flush enter 2024-03-29T11:13:50.691 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.713+0000 7fee25638640 10 librados: watch_flush exit 2024-03-29T11:13:50.691 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.713+0000 7fee25638640 20 client.4264.objecter shutdown clearing up homeless session... 2024-03-29T11:13:50.691 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.713+0000 7fee25638640 10 client.4264.objecter successfully canceled tick 2024-03-29T11:13:50.691 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.713+0000 7fee25638640 1 -- 172.21.15.47:0/2994338131 >> v1:172.21.15.121:6816/1964117370 conn(0x7fee0c06e500 legacy=0x7fee0c0709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:50.691 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.713+0000 7fee25638640 10 monclient: shutdown 2024-03-29T11:13:50.691 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.713+0000 7fee25638640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:50.691 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.713+0000 7fee25638640 1 -- 172.21.15.47:0/2994338131 >> v1:172.21.15.121:6789/0 conn(0x7fee200b0770 legacy=0x7fee20146ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:50.695 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.717+0000 7fee25638640 1 -- 172.21.15.47:0/2994338131 shutdown_connections 2024-03-29T11:13:50.695 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.717+0000 7fee25638640 1 -- 172.21.15.47:0/2994338131 >> 172.21.15.47:0/2994338131 conn(0x7fee2006b660 msgr2=0x7fee2006ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:50.695 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.717+0000 7fee25638640 1 -- 172.21.15.47:0/2994338131 shutdown_connections 2024-03-29T11:13:50.695 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.717+0000 7fee25638640 1 -- 172.21.15.47:0/2994338131 wait complete. 2024-03-29T11:13:50.695 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:50.717+0000 7fee25638640 1 librados: shutdown 2024-03-29T11:13:50.711 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:13:57.518 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:13:57.674 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.696+0000 7f95c841e640 10 monclient: get_monmap_and_config 2024-03-29T11:13:57.674 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.696+0000 7f95c841e640 10 monclient: build_initial_monmap 2024-03-29T11:13:57.675 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.696+0000 7f95c841e640 10 monclient: monmap: 2024-03-29T11:13:57.675 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:13:57.675 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:13:57.675 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:13:57.697830+0000 2024-03-29T11:13:57.675 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:13:57.697830+0000 2024-03-29T11:13:57.675 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:57.675 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:13:57.675 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:13:57.675 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:13:57.675 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:13:57.675 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:13:57.676 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.697+0000 7f95c841e640 1 Processor -- start 2024-03-29T11:13:57.676 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.697+0000 7f95c841e640 1 -- start start 2024-03-29T11:13:57.676 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.697+0000 7f95c841e640 10 monclient: init 2024-03-29T11:13:57.677 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.698+0000 7f95c841e640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:57.678 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.698+0000 7f95c841e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:13:57.678 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.698+0000 7f95c841e640 10 monclient(hunting): picked mon.noname-c con 0x7f95c00b0860 addr v1:172.21.15.121:6789/0 2024-03-29T11:13:57.678 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.698+0000 7f95c841e640 10 monclient(hunting): picked mon.noname-b con 0x7f95c013ef30 addr v1:172.21.15.47:6790/0 2024-03-29T11:13:57.678 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.698+0000 7f95c841e640 10 monclient(hunting): picked mon.noname-a con 0x7f95c00af8e0 addr v1:172.21.15.47:6789/0 2024-03-29T11:13:57.678 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.698+0000 7f95c841e640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95c00b0c40 con 0x7f95c00af8e0 2024-03-29T11:13:57.678 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.698+0000 7f95c841e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95c00aed10 con 0x7f95c00b0860 2024-03-29T11:13:57.678 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.698+0000 7f95c841e640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95c00afd50 con 0x7f95c013ef30 2024-03-29T11:13:57.678 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.698+0000 7f95c841e640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:57.678 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.698+0000 7f95c841e640 10 monclient(hunting): authenticate will time out at 880.886889s 2024-03-29T11:13:57.678 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.699+0000 7f95c69bb640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f95c00af8e0 0x7f95c0149a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:42404/0 (socket says 172.21.15.47:42404) 2024-03-29T11:13:57.678 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.699+0000 7f95c69bb640 1 -- 172.21.15.47:0/1835694062 learned_addr learned my addr 172.21.15.47:0/1835694062 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:13:57.678 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.699+0000 7f95c51b8640 1 -- 172.21.15.47:0/1835694062 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3176613264 0 0) 0x7f95c00aed10 con 0x7f95c00b0860 2024-03-29T11:13:57.679 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.699+0000 7f95c51b8640 10 monclient(hunting): _negotiate 2024-03-29T11:13:57.679 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.699+0000 7f95c51b8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:57.679 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.699+0000 7f95c51b8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:57.679 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.699+0000 7f95c51b8640 10 monclient(hunting): my global_id is 4267 2024-03-29T11:13:57.679 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.699+0000 7f95c51b8640 1 -- 172.21.15.47:0/1835694062 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95ac003520 con 0x7f95c00b0860 2024-03-29T11:13:57.679 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.700+0000 7f95c51b8640 1 -- 172.21.15.47:0/1835694062 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2278261917 0 0) 0x7f95c00b0c40 con 0x7f95c00af8e0 2024-03-29T11:13:57.679 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.700+0000 7f95c51b8640 10 monclient(hunting): _negotiate 2024-03-29T11:13:57.679 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.700+0000 7f95c51b8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:57.679 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.700+0000 7f95c51b8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:57.679 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.700+0000 7f95c51b8640 10 monclient(hunting): my global_id is 4275 2024-03-29T11:13:57.679 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.700+0000 7f95c51b8640 1 -- 172.21.15.47:0/1835694062 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95c00aed10 con 0x7f95c00af8e0 2024-03-29T11:13:57.692 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.700+0000 7f95c51b8640 1 -- 172.21.15.47:0/1835694062 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4098655328 0 0) 0x7f95ac003520 con 0x7f95c00b0860 2024-03-29T11:13:57.692 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.700+0000 7f95c51b8640 1 -- 172.21.15.47:0/1835694062 >> v1:172.21.15.47:6790/0 conn(0x7f95c013ef30 legacy=0x7f95c00af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:57.692 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.700+0000 7f95c51b8640 1 -- 172.21.15.47:0/1835694062 >> v1:172.21.15.47:6789/0 conn(0x7f95c00af8e0 legacy=0x7f95c0149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:57.692 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.700+0000 7f95c51b8640 10 monclient: _finish_hunting 0 2024-03-29T11:13:57.692 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.700+0000 7f95c51b8640 1 monclient: found mon.noname-c 2024-03-29T11:13:57.692 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.700+0000 7f95c51b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:57.692 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.700+0000 7f95c51b8640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:13:57.692 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.700+0000 7f95c51b8640 1 -- 172.21.15.47:0/1835694062 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95c00ab1b0 con 0x7f95c00b0860 2024-03-29T11:13:57.693 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.700+0000 7f95c51b8640 10 monclient: _finish_auth 0 2024-03-29T11:13:57.693 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.700+0000 7f95c51b8640 10 monclient: _check_auth_tickets 2024-03-29T11:13:57.693 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.700+0000 7f95c51b8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:57.693 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.700+0000 7f95c51b8640 1 -- 172.21.15.47:0/1835694062 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f95b8001a10 con 0x7f95c00b0860 2024-03-29T11:13:57.693 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.700+0000 7f95c51b8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:57.693 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.700+0000 7f95c51b8640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:13:57.693 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.700+0000 7f95c51b8640 10 monclient: dump: 2024-03-29T11:13:57.693 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:13:57.693 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:13:57.693 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:57.693 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:57.693 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:57.693 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:13:57.693 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:13:57.693 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:13:57.694 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:13:57.694 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:13:57.694 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.700+0000 7f95c841e640 5 monclient: authenticate success, global_id 4267 2024-03-29T11:13:57.694 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.700+0000 7f95c841e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:13:57.694 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.701+0000 7f95c51b8640 1 -- 172.21.15.47:0/1835694062 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f95b8001c30 con 0x7f95c00b0860 2024-03-29T11:13:57.694 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.701+0000 7f95c51b8640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:57.695 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.701+0000 7f95c841e640 10 monclient: get_monmap_and_config success 2024-03-29T11:13:57.695 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.701+0000 7f95c841e640 4 set_mon_vals no callback set 2024-03-29T11:13:57.695 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.702+0000 7f95c841e640 10 monclient: shutdown 2024-03-29T11:13:57.695 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.702+0000 7f95c841e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:57.695 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.702+0000 7f95c841e640 1 -- 172.21.15.47:0/1835694062 >> v1:172.21.15.121:6789/0 conn(0x7f95c00b0860 legacy=0x7f95c013e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:57.695 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.702+0000 7f95c841e640 1 -- 172.21.15.47:0/1835694062 shutdown_connections 2024-03-29T11:13:57.695 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.702+0000 7f95c841e640 1 -- 172.21.15.47:0/1835694062 >> 172.21.15.47:0/1835694062 conn(0x7f95c006b660 msgr2=0x7f95c006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:57.696 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.702+0000 7f95c841e640 1 -- 172.21.15.47:0/1835694062 shutdown_connections 2024-03-29T11:13:57.696 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.702+0000 7f95c841e640 1 -- 172.21.15.47:0/1835694062 wait complete. 2024-03-29T11:13:57.696 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.703+0000 7f95c841e640 10 monclient: build_initial_monmap 2024-03-29T11:13:57.696 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.703+0000 7f95c841e640 10 monclient: monmap: 2024-03-29T11:13:57.696 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:13:57.696 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:13:57.696 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:13:57.696 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:13:57.696 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:57.696 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:13:57.696 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:13:57.696 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:13:57.696 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:13:57.696 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:13:57.696 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.704+0000 7f95c841e640 1 librados: starting msgr at 2024-03-29T11:13:57.697 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.704+0000 7f95c841e640 1 librados: starting objecter 2024-03-29T11:13:57.697 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.704+0000 7f95c841e640 1 Processor -- start 2024-03-29T11:13:57.697 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.704+0000 7f95c841e640 1 -- start start 2024-03-29T11:13:57.697 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.704+0000 7f95c841e640 1 librados: setting wanted keys 2024-03-29T11:13:57.697 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.704+0000 7f95c841e640 1 librados: calling monclient init 2024-03-29T11:13:57.697 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.704+0000 7f95c841e640 10 monclient: init 2024-03-29T11:13:57.697 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.705+0000 7f95c841e640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:57.697 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.705+0000 7f95c841e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:13:57.697 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.705+0000 7f95c841e640 10 monclient(hunting): picked mon.noname-a con 0x7f95c00af8e0 addr v1:172.21.15.47:6789/0 2024-03-29T11:13:57.697 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.705+0000 7f95c61ba640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f95c00af8e0 0x7f95c0070e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:42518/0 (socket says 172.21.15.47:42518) 2024-03-29T11:13:57.697 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.705+0000 7f95c59b9640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f95c00b0860 0x7f95c006e7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:56174/0 (socket says 172.21.15.47:56174) 2024-03-29T11:13:57.697 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.705+0000 7f95c841e640 10 monclient(hunting): picked mon.noname-c con 0x7f95c00b0860 addr v1:172.21.15.47:6790/0 2024-03-29T11:13:57.697 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.706+0000 7f95c59b9640 1 -- 172.21.15.47:0/1937466914 learned_addr learned my addr 172.21.15.47:0/1937466914 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:13:57.698 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.706+0000 7f95c841e640 10 monclient(hunting): picked mon.noname-b con 0x7f95c013ef30 addr v1:172.21.15.121:6789/0 2024-03-29T11:13:57.698 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.706+0000 7f95c841e640 1 -- 172.21.15.47:0/1937466914 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95c006e310 con 0x7f95c00af8e0 2024-03-29T11:13:57.698 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.706+0000 7f95b6ffd640 10 client.?.objecter ms_handle_connect 0x7f95c00b0860 2024-03-29T11:13:57.698 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.706+0000 7f95b6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:57.698 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.706+0000 7f95b6ffd640 10 client.?.objecter ms_handle_connect 0x7f95c00af8e0 2024-03-29T11:13:57.698 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.706+0000 7f95b6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:57.698 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.706+0000 7f95c841e640 1 -- 172.21.15.47:0/1937466914 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95c0070700 con 0x7f95c013ef30 2024-03-29T11:13:57.698 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.706+0000 7f95c841e640 1 -- 172.21.15.47:0/1937466914 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95c00709e0 con 0x7f95c00b0860 2024-03-29T11:13:57.699 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.706+0000 7f95c841e640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:57.699 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.706+0000 7f95c841e640 10 monclient(hunting): authenticate will time out at 880.894527s 2024-03-29T11:13:57.699 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.706+0000 7f95b6ffd640 10 client.?.objecter ms_handle_connect 0x7f95c013ef30 2024-03-29T11:13:57.699 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.706+0000 7f95b6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:57.699 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.706+0000 7f95b6ffd640 1 -- 172.21.15.47:0/1937466914 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3765194872 0 0) 0x7f95c006e310 con 0x7f95c00af8e0 2024-03-29T11:13:57.699 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.706+0000 7f95b6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:13:57.699 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.707+0000 7f95b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:57.699 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.707+0000 7f95b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:57.699 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.707+0000 7f95b6ffd640 10 monclient(hunting): my global_id is 4278 2024-03-29T11:13:57.699 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.707+0000 7f95b6ffd640 1 -- 172.21.15.47:0/1937466914 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95a4004eb0 con 0x7f95c00af8e0 2024-03-29T11:13:57.699 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.707+0000 7f95b6ffd640 1 -- 172.21.15.47:0/1937466914 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3260815764 0 0) 0x7f95c0070700 con 0x7f95c013ef30 2024-03-29T11:13:57.699 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.707+0000 7f95b6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:13:57.699 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.707+0000 7f95b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:57.700 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.707+0000 7f95b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:57.700 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.707+0000 7f95b6ffd640 10 monclient(hunting): my global_id is 4270 2024-03-29T11:13:57.700 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.707+0000 7f95b6ffd640 1 -- 172.21.15.47:0/1937466914 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95c006e310 con 0x7f95c013ef30 2024-03-29T11:13:57.700 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.708+0000 7f95b6ffd640 1 -- 172.21.15.47:0/1937466914 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 195183228 0 0) 0x7f95a4004eb0 con 0x7f95c00af8e0 2024-03-29T11:13:57.700 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.708+0000 7f95b6ffd640 1 -- 172.21.15.47:0/1937466914 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95c0070700 con 0x7f95c00af8e0 2024-03-29T11:13:57.700 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.708+0000 7f95b6ffd640 1 -- 172.21.15.47:0/1937466914 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f95b8003b10 con 0x7f95c00af8e0 2024-03-29T11:13:57.700 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.708+0000 7f95b6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:57.700 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.708+0000 7f95b6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:13:57.700 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.708+0000 7f95b6ffd640 10 monclient(hunting): dump: 2024-03-29T11:13:57.700 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:13:57.700 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:13:57.700 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:57.700 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:57.700 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:57.700 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:13:57.701 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:13:57.701 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:13:57.701 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:13:57.701 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:13:57.701 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.708+0000 7f95b6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:13:57.701 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.708+0000 7f95b6ffd640 1 -- 172.21.15.47:0/1937466914 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3878143212 0 0) 0x7f95c006e310 con 0x7f95c013ef30 2024-03-29T11:13:57.701 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.708+0000 7f95b6ffd640 1 -- 172.21.15.47:0/1937466914 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95a4004eb0 con 0x7f95c013ef30 2024-03-29T11:13:57.701 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.708+0000 7f95b6ffd640 1 -- 172.21.15.47:0/1937466914 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1343289321 0 0) 0x7f95c00709e0 con 0x7f95c00b0860 2024-03-29T11:13:57.702 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.708+0000 7f95b6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:13:57.702 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.708+0000 7f95b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:57.702 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.708+0000 7f95b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:57.702 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.708+0000 7f95b6ffd640 10 monclient(hunting): my global_id is 4280 2024-03-29T11:13:57.702 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.708+0000 7f95b6ffd640 1 -- 172.21.15.47:0/1937466914 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95c006e310 con 0x7f95c00b0860 2024-03-29T11:13:57.702 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.708+0000 7f95b6ffd640 1 -- 172.21.15.47:0/1937466914 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f95b0001930 con 0x7f95c013ef30 2024-03-29T11:13:57.702 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.708+0000 7f95b6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:57.702 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.708+0000 7f95b6ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:13:57.702 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.708+0000 7f95b6ffd640 10 monclient(hunting): dump: 2024-03-29T11:13:57.702 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:13:57.702 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:13:57.702 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:57.702 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:57.703 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:57.703 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:13:57.703 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:13:57.703 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:13:57.703 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:13:57.703 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:13:57.703 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.708+0000 7f95b6ffd640 1 -- 172.21.15.47:0/1937466914 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3552354573 0 0) 0x7f95a4004eb0 con 0x7f95c013ef30 2024-03-29T11:13:57.703 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.709+0000 7f95b6ffd640 1 -- 172.21.15.47:0/1937466914 >> v1:172.21.15.47:6790/0 conn(0x7f95c00b0860 legacy=0x7f95c006e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:57.703 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.709+0000 7f95b6ffd640 1 -- 172.21.15.47:0/1937466914 >> v1:172.21.15.47:6789/0 conn(0x7f95c00af8e0 legacy=0x7f95c0070e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:57.703 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.709+0000 7f95b6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:13:57.703 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.709+0000 7f95b6ffd640 1 monclient: found mon.b 2024-03-29T11:13:57.703 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.709+0000 7f95b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:57.703 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.709+0000 7f95b6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:13:57.703 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.709+0000 7f95b6ffd640 1 -- 172.21.15.47:0/1937466914 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95c006f940 con 0x7f95c013ef30 2024-03-29T11:13:57.703 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.709+0000 7f95b6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:13:57.704 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.709+0000 7f95b6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:13:57.704 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.709+0000 7f95b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:57.704 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.709+0000 7f95c841e640 5 monclient: authenticate success, global_id 4270 2024-03-29T11:13:57.704 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.709+0000 7f95c841e640 10 monclient: _renew_subs 2024-03-29T11:13:57.704 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.709+0000 7f95c841e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:13:57.704 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.709+0000 7f95c841e640 1 -- 172.21.15.47:0/1937466914 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95c0133cf0 con 0x7f95c013ef30 2024-03-29T11:13:57.704 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.709+0000 7f95c841e640 10 client.4270.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:13:57.704 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.709+0000 7f95c841e640 10 monclient: _renew_subs 2024-03-29T11:13:57.704 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.709+0000 7f95c841e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:13:57.704 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.709+0000 7f95c841e640 1 -- 172.21.15.47:0/1937466914 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f95c01331f0 con 0x7f95c013ef30 2024-03-29T11:13:57.704 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.709+0000 7f95c841e640 1 librados: init done 2024-03-29T11:13:57.704 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.710+0000 7f95b6ffd640 1 -- 172.21.15.47:0/1937466914 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f95b00047e0 con 0x7f95c013ef30 2024-03-29T11:13:57.705 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.710+0000 7f95b6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:57.705 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.710+0000 7f95b6ffd640 1 -- 172.21.15.47:0/1937466914 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f95b0004af0 con 0x7f95c013ef30 2024-03-29T11:13:57.705 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.710+0000 7f95c49b7640 4 set_mon_vals no callback set 2024-03-29T11:13:57.705 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.710+0000 7f95b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:57.705 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.710+0000 7f95b6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:13:57.706 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.710+0000 7f95b6ffd640 10 monclient: dump: 2024-03-29T11:13:57.706 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:13:57.706 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:13:57.706 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:57.706 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:13:57.706 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:57.706 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:13:57.706 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:13:57.706 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:13:57.706 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:13:57.706 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:13:57.706 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.711+0000 7f95c841e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:57.706 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.712+0000 7f95b6ffd640 1 -- 172.21.15.47:0/1937466914 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7f95b0004c50 con 0x7f95c013ef30 2024-03-29T11:13:57.706 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.713+0000 7f95c841e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:57.706 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.713+0000 7f95c841e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:13:57.707 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.713+0000 7f95c841e640 1 -- 172.21.15.47:0/1937466914 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95a8005080 con 0x7f95c013ef30 2024-03-29T11:13:57.707 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.713+0000 7f95c61ba640 10 monclient: get_auth_request con 0x7f95a406df50 auth_method 0 2024-03-29T11:13:57.707 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.713+0000 7f95b6ffd640 1 -- 172.21.15.47:0/1937466914 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7f95b00848a0 con 0x7f95c013ef30 2024-03-29T11:13:57.707 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.713+0000 7f95b6ffd640 10 client.4270.objecter ms_dispatch 0x7f95c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:13:57.707 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.713+0000 7f95b6ffd640 3 client.4270.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:13:57.707 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.713+0000 7f95b6ffd640 3 client.4270.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:13:57.707 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.713+0000 7f95b6ffd640 20 client.4270.objecter dump_active .. 0 homeless 2024-03-29T11:13:57.707 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.713+0000 7f95b6ffd640 10 client.4270.objecter ms_handle_connect 0x7f95a406df50 2024-03-29T11:13:57.708 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.719+0000 7f95b6ffd640 1 -- 172.21.15.47:0/1937466914 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f95b0050180 con 0x7f95c013ef30 2024-03-29T11:13:57.708 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.720+0000 7f95b6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:57.708 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:57.720+0000 7f95b6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:13:58.131 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:58.153+0000 7f95c841e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:58.131 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:58.153+0000 7f95c841e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:58.131 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:58.153+0000 7f95c841e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:13:58.131 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:58.153+0000 7f95c841e640 1 -- 172.21.15.47:0/1937466914 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f95a8005330 con 0x7f95c013ef30 2024-03-29T11:13:58.132 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:58.154+0000 7f95b6ffd640 1 -- 172.21.15.47:0/1937466914 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7f95b00520b0 con 0x7f95c013ef30 2024-03-29T11:13:58.132 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:58.154+0000 7f95b6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:58.132 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:58.154+0000 7f95b6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:13:58.132 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:13:58.132 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:13:58.136 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:58.158+0000 7f95c841e640 10 librados: watch_flush enter 2024-03-29T11:13:58.136 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:58.158+0000 7f95c841e640 10 librados: watch_flush exit 2024-03-29T11:13:58.136 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:58.158+0000 7f95c841e640 20 client.4270.objecter shutdown clearing up homeless session... 2024-03-29T11:13:58.136 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:58.158+0000 7f95c841e640 10 client.4270.objecter successfully canceled tick 2024-03-29T11:13:58.137 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:58.158+0000 7f95c841e640 1 -- 172.21.15.47:0/1937466914 >> v1:172.21.15.121:6816/1964117370 conn(0x7f95a406df50 legacy=0x7f95a40703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:58.137 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:58.158+0000 7f95c841e640 10 monclient: shutdown 2024-03-29T11:13:58.137 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:58.158+0000 7f95c841e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:58.137 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:58.158+0000 7f95c841e640 1 -- 172.21.15.47:0/1937466914 >> v1:172.21.15.121:6789/0 conn(0x7f95c013ef30 legacy=0x7f95c006f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:58.137 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:58.158+0000 7f95c841e640 1 -- 172.21.15.47:0/1937466914 shutdown_connections 2024-03-29T11:13:58.137 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:58.158+0000 7f95c841e640 1 -- 172.21.15.47:0/1937466914 >> 172.21.15.47:0/1937466914 conn(0x7f95c006b660 msgr2=0x7f95c013e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:58.137 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:58.159+0000 7f95c841e640 1 -- 172.21.15.47:0/1937466914 shutdown_connections 2024-03-29T11:13:58.138 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:58.159+0000 7f95c841e640 1 -- 172.21.15.47:0/1937466914 wait complete. 2024-03-29T11:13:58.138 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:13:58.159+0000 7f95c841e640 1 librados: shutdown 2024-03-29T11:13:58.153 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:14:04.959 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:14:05.096 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.118+0000 7fa328d10640 10 monclient: get_monmap_and_config 2024-03-29T11:14:05.096 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.118+0000 7fa328d10640 10 monclient: build_initial_monmap 2024-03-29T11:14:05.096 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.118+0000 7fa328d10640 10 monclient: monmap: 2024-03-29T11:14:05.096 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:14:05.096 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:14:05.096 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:14:05.119263+0000 2024-03-29T11:14:05.096 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:14:05.119263+0000 2024-03-29T11:14:05.096 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:05.096 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:14:05.096 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:14:05.096 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:14:05.096 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:14:05.097 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:14:05.097 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.119+0000 7fa328d10640 1 Processor -- start 2024-03-29T11:14:05.097 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.119+0000 7fa328d10640 1 -- start start 2024-03-29T11:14:05.097 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.119+0000 7fa328d10640 10 monclient: init 2024-03-29T11:14:05.098 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.120+0000 7fa328d10640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:05.098 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.120+0000 7fa328d10640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:14:05.098 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.120+0000 7fa328d10640 10 monclient(hunting): picked mon.noname-b con 0x7fa3241457f0 addr v1:172.21.15.47:6790/0 2024-03-29T11:14:05.098 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.120+0000 7fa328d10640 10 monclient(hunting): picked mon.noname-c con 0x7fa32413f650 addr v1:172.21.15.121:6789/0 2024-03-29T11:14:05.098 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.120+0000 7fa328d10640 10 monclient(hunting): picked mon.noname-a con 0x7fa3240a9d70 addr v1:172.21.15.47:6789/0 2024-03-29T11:14:05.098 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.120+0000 7fa328d10640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3240aaef0 con 0x7fa3240a9d70 2024-03-29T11:14:05.098 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.120+0000 7fa328d10640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa324145bd0 con 0x7fa32413f650 2024-03-29T11:14:05.098 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.120+0000 7fa328d10640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa324145eb0 con 0x7fa3241457f0 2024-03-29T11:14:05.098 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.120+0000 7fa328d10640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:05.098 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.120+0000 7fa328d10640 10 monclient(hunting): authenticate will time out at 888.308268s 2024-03-29T11:14:05.099 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.121+0000 7fa32259c640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fa3241457f0 0x7fa32413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:43764/0 (socket says 172.21.15.47:43764) 2024-03-29T11:14:05.099 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.121+0000 7fa32259c640 1 -- 172.21.15.47:0/2416649676 learned_addr learned my addr 172.21.15.47:0/2416649676 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:14:05.100 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.121+0000 7fa321d9b640 1 -- 172.21.15.47:0/2416649676 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1046708344 0 0) 0x7fa324145eb0 con 0x7fa3241457f0 2024-03-29T11:14:05.100 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.121+0000 7fa321d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:14:05.100 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.121+0000 7fa321d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:05.100 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.121+0000 7fa321d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:05.100 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.121+0000 7fa321d9b640 10 monclient(hunting): my global_id is 4283 2024-03-29T11:14:05.100 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.121+0000 7fa321d9b640 1 -- 172.21.15.47:0/2416649676 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa310003520 con 0x7fa3241457f0 2024-03-29T11:14:05.100 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.121+0000 7fa321d9b640 1 -- 172.21.15.47:0/2416649676 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1833874009 0 0) 0x7fa310003520 con 0x7fa3241457f0 2024-03-29T11:14:05.100 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.121+0000 7fa321d9b640 1 -- 172.21.15.47:0/2416649676 >> v1:172.21.15.121:6789/0 conn(0x7fa32413f650 legacy=0x7fa3240b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:14:05.100 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.121+0000 7fa321d9b640 1 -- 172.21.15.47:0/2416649676 >> v1:172.21.15.47:6789/0 conn(0x7fa3240a9d70 legacy=0x7fa32414d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:14:05.103 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.123+0000 7fa321d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:14:05.104 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.123+0000 7fa321d9b640 1 monclient: found mon.noname-b 2024-03-29T11:14:05.104 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.123+0000 7fa321d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:05.104 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.123+0000 7fa321d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-29T11:14:05.104 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.123+0000 7fa321d9b640 1 -- 172.21.15.47:0/2416649676 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa324146340 con 0x7fa3241457f0 2024-03-29T11:14:05.104 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.123+0000 7fa321d9b640 10 monclient: _finish_auth 0 2024-03-29T11:14:05.104 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.123+0000 7fa321d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:14:05.104 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.123+0000 7fa321d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:05.104 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.123+0000 7fa321d9b640 1 -- 172.21.15.47:0/2416649676 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fa314001690 con 0x7fa3241457f0 2024-03-29T11:14:05.106 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.123+0000 7fa328d10640 5 monclient: authenticate success, global_id 4283 2024-03-29T11:14:05.106 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.123+0000 7fa328d10640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:05.106 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.124+0000 7fa321d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:05.106 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.124+0000 7fa321d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:14:05.106 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.124+0000 7fa321d9b640 10 monclient: dump: 2024-03-29T11:14:05.106 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:14:05.106 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:14:05.106 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:05.106 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:05.106 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:05.106 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:14:05.107 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:14:05.107 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:14:05.107 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:14:05.107 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:14:05.107 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.124+0000 7fa328d10640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:05.107 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.124+0000 7fa321d9b640 1 -- 172.21.15.47:0/2416649676 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa314001920 con 0x7fa3241457f0 2024-03-29T11:14:05.107 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.124+0000 7fa321d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:05.107 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.124+0000 7fa328d10640 10 monclient: get_monmap_and_config success 2024-03-29T11:14:05.107 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.124+0000 7fa328d10640 4 set_mon_vals no callback set 2024-03-29T11:14:05.107 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.124+0000 7fa321d9b640 1 -- 172.21.15.47:0/2416649676 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fa314001cb0 con 0x7fa3241457f0 2024-03-29T11:14:05.107 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.124+0000 7fa321d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:05.107 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.124+0000 7fa321d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:14:05.107 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.124+0000 7fa321d9b640 10 monclient: dump: 2024-03-29T11:14:05.107 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:14:05.107 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:14:05.108 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:05.108 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:05.108 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:05.108 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:14:05.108 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:14:05.108 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:14:05.108 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:14:05.108 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:14:05.108 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.125+0000 7fa328d10640 10 monclient: shutdown 2024-03-29T11:14:05.108 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.125+0000 7fa328d10640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:05.108 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.125+0000 7fa328d10640 1 -- 172.21.15.47:0/2416649676 >> v1:172.21.15.47:6790/0 conn(0x7fa3241457f0 legacy=0x7fa32413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:05.108 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.125+0000 7fa328d10640 1 -- 172.21.15.47:0/2416649676 shutdown_connections 2024-03-29T11:14:05.108 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.125+0000 7fa328d10640 1 -- 172.21.15.47:0/2416649676 >> 172.21.15.47:0/2416649676 conn(0x7fa324066a00 msgr2=0x7fa324066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:05.108 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.125+0000 7fa328d10640 1 -- 172.21.15.47:0/2416649676 shutdown_connections 2024-03-29T11:14:05.108 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.125+0000 7fa328d10640 1 -- 172.21.15.47:0/2416649676 wait complete. 2024-03-29T11:14:05.109 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.126+0000 7fa328d10640 10 monclient: build_initial_monmap 2024-03-29T11:14:05.109 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.126+0000 7fa328d10640 10 monclient: monmap: 2024-03-29T11:14:05.109 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:14:05.109 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:14:05.109 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:14:05.109 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:14:05.109 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:05.109 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:14:05.109 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:14:05.109 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:14:05.109 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:14:05.109 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:14:05.117 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.127+0000 7fa328d10640 1 librados: starting msgr at 2024-03-29T11:14:05.117 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.127+0000 7fa328d10640 1 librados: starting objecter 2024-03-29T11:14:05.117 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.127+0000 7fa328d10640 1 Processor -- start 2024-03-29T11:14:05.117 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.127+0000 7fa328d10640 1 -- start start 2024-03-29T11:14:05.117 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.127+0000 7fa328d10640 1 librados: setting wanted keys 2024-03-29T11:14:05.117 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.127+0000 7fa328d10640 1 librados: calling monclient init 2024-03-29T11:14:05.117 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.127+0000 7fa328d10640 10 monclient: init 2024-03-29T11:14:05.117 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.128+0000 7fa328d10640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:05.117 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.128+0000 7fa328d10640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:14:05.117 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.128+0000 7fa328d10640 10 monclient(hunting): picked mon.noname-a con 0x7fa3240a9d70 addr v1:172.21.15.47:6789/0 2024-03-29T11:14:05.117 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.128+0000 7fa328d10640 10 monclient(hunting): picked mon.noname-b con 0x7fa32413f650 addr v1:172.21.15.121:6789/0 2024-03-29T11:14:05.117 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.128+0000 7fa31bfff640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fa32413f650 0x7fa3241348c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.47:58848/0 (socket says 172.21.15.47:58848) 2024-03-29T11:14:05.118 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.128+0000 7fa322d9d640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fa3241457f0 0x7fa32413adf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:44044/0 (socket says 172.21.15.47:44044) 2024-03-29T11:14:05.118 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.128+0000 7fa322d9d640 1 -- 172.21.15.47:0/2472307793 learned_addr learned my addr 172.21.15.47:0/2472307793 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:14:05.118 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.129+0000 7fa328d10640 10 monclient(hunting): picked mon.noname-c con 0x7fa3241457f0 addr v1:172.21.15.47:6790/0 2024-03-29T11:14:05.118 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.129+0000 7fa31affd640 10 client.?.objecter ms_handle_connect 0x7fa3241457f0 2024-03-29T11:14:05.118 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.129+0000 7fa31affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:05.118 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.129+0000 7fa328d10640 1 -- 172.21.15.47:0/2472307793 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa324135d40 con 0x7fa3240a9d70 2024-03-29T11:14:05.118 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.129+0000 7fa31affd640 10 client.?.objecter ms_handle_connect 0x7fa32413f650 2024-03-29T11:14:05.118 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.129+0000 7fa31affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:05.118 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.129+0000 7fa328d10640 1 -- 172.21.15.47:0/2472307793 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa324136470 con 0x7fa32413f650 2024-03-29T11:14:05.118 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.129+0000 7fa328d10640 1 -- 172.21.15.47:0/2472307793 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa324136600 con 0x7fa3241457f0 2024-03-29T11:14:05.118 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.129+0000 7fa328d10640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:05.118 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.129+0000 7fa328d10640 10 monclient(hunting): authenticate will time out at 888.317176s 2024-03-29T11:14:05.118 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.129+0000 7fa31affd640 1 -- 172.21.15.47:0/2472307793 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3636897732 0 0) 0x7fa324136600 con 0x7fa3241457f0 2024-03-29T11:14:05.119 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.129+0000 7fa31affd640 10 monclient(hunting): _negotiate 2024-03-29T11:14:05.119 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.129+0000 7fa31affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:05.119 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.129+0000 7fa31affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:05.119 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.129+0000 7fa31affd640 10 monclient(hunting): my global_id is 4286 2024-03-29T11:14:05.119 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.129+0000 7fa31affd640 1 -- 172.21.15.47:0/2472307793 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa304004eb0 con 0x7fa3241457f0 2024-03-29T11:14:05.119 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.129+0000 7fa31affd640 1 -- 172.21.15.47:0/2472307793 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 406452971 0 0) 0x7fa324136470 con 0x7fa32413f650 2024-03-29T11:14:05.119 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.129+0000 7fa31affd640 10 monclient(hunting): _negotiate 2024-03-29T11:14:05.119 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.129+0000 7fa31affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:05.119 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.129+0000 7fa31affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:05.119 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.129+0000 7fa31affd640 10 monclient(hunting): my global_id is 4273 2024-03-29T11:14:05.119 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.129+0000 7fa31affd640 1 -- 172.21.15.47:0/2472307793 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa324136600 con 0x7fa32413f650 2024-03-29T11:14:05.119 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.129+0000 7fa31affd640 1 -- 172.21.15.47:0/2472307793 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3109586748 0 0) 0x7fa304004eb0 con 0x7fa3241457f0 2024-03-29T11:14:05.119 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.129+0000 7fa31affd640 1 -- 172.21.15.47:0/2472307793 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa324136470 con 0x7fa3241457f0 2024-03-29T11:14:05.119 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.129+0000 7fa31affd640 1 -- 172.21.15.47:0/2472307793 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fa31c001c60 con 0x7fa3241457f0 2024-03-29T11:14:05.119 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.129+0000 7fa31affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:05.120 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.129+0000 7fa31affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:14:05.120 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.129+0000 7fa31affd640 10 monclient(hunting): dump: 2024-03-29T11:14:05.120 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:14:05.120 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:14:05.120 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:05.120 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:05.120 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:05.120 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:14:05.120 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:14:05.120 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:14:05.120 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:14:05.120 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:14:05.120 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.129+0000 7fa31affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:14:05.120 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.129+0000 7fa31affd640 1 -- 172.21.15.47:0/2472307793 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3882044534 0 0) 0x7fa324136600 con 0x7fa32413f650 2024-03-29T11:14:05.121 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.129+0000 7fa31affd640 1 -- 172.21.15.47:0/2472307793 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa304004eb0 con 0x7fa32413f650 2024-03-29T11:14:05.121 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.129+0000 7fa31affd640 1 -- 172.21.15.47:0/2472307793 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fa30c001860 con 0x7fa32413f650 2024-03-29T11:14:05.121 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.129+0000 7fa31affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:05.121 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.129+0000 7fa31affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:14:05.121 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.129+0000 7fa31affd640 10 monclient(hunting): dump: 2024-03-29T11:14:05.121 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:14:05.121 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:14:05.121 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:05.121 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:05.121 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:05.121 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:14:05.121 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:14:05.121 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:14:05.121 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:14:05.121 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:14:05.122 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.130+0000 7fa31affd640 1 -- 172.21.15.47:0/2472307793 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1913516937 0 0) 0x7fa324136470 con 0x7fa3241457f0 2024-03-29T11:14:05.122 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.130+0000 7fa31affd640 1 -- 172.21.15.47:0/2472307793 >> v1:172.21.15.121:6789/0 conn(0x7fa32413f650 legacy=0x7fa3241348c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:05.122 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.130+0000 7fa31affd640 1 -- 172.21.15.47:0/2472307793 >> v1:172.21.15.47:6789/0 conn(0x7fa3240a9d70 legacy=0x7fa3241da560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:14:05.122 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.131+0000 7fa31affd640 10 monclient: _finish_hunting 0 2024-03-29T11:14:05.122 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.131+0000 7fa31affd640 1 monclient: found mon.c 2024-03-29T11:14:05.122 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.131+0000 7fa31affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:05.122 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.131+0000 7fa31affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:14:05.122 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.131+0000 7fa31affd640 1 -- 172.21.15.47:0/2472307793 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa32413b500 con 0x7fa3241457f0 2024-03-29T11:14:05.127 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.131+0000 7fa31affd640 10 monclient: _finish_auth 0 2024-03-29T11:14:05.127 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.131+0000 7fa31affd640 10 monclient: _check_auth_tickets 2024-03-29T11:14:05.127 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.131+0000 7fa31affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:05.127 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.131+0000 7fa328d10640 5 monclient: authenticate success, global_id 4286 2024-03-29T11:14:05.127 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.131+0000 7fa328d10640 10 monclient: _renew_subs 2024-03-29T11:14:05.127 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.131+0000 7fa328d10640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:14:05.127 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.131+0000 7fa328d10640 1 -- 172.21.15.47:0/2472307793 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa32413b7b0 con 0x7fa3241457f0 2024-03-29T11:14:05.127 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.131+0000 7fa328d10640 10 client.4286.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:14:05.127 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.131+0000 7fa328d10640 10 monclient: _renew_subs 2024-03-29T11:14:05.127 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.131+0000 7fa328d10640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:14:05.127 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.131+0000 7fa328d10640 1 -- 172.21.15.47:0/2472307793 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa324142860 con 0x7fa3241457f0 2024-03-29T11:14:05.127 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.131+0000 7fa31affd640 1 -- 172.21.15.47:0/2472307793 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa31c003970 con 0x7fa3241457f0 2024-03-29T11:14:05.128 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.131+0000 7fa328d10640 1 librados: init done 2024-03-29T11:14:05.128 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.131+0000 7fa31affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:05.128 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.131+0000 7fa31affd640 1 -- 172.21.15.47:0/2472307793 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fa31c003d40 con 0x7fa3241457f0 2024-03-29T11:14:05.128 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.131+0000 7fa31affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:05.128 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.131+0000 7fa32159a640 4 set_mon_vals no callback set 2024-03-29T11:14:05.128 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.131+0000 7fa31affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:14:05.128 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.131+0000 7fa31affd640 10 monclient: dump: 2024-03-29T11:14:05.128 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:14:05.128 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:14:05.128 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:05.128 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:05.128 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:05.128 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:14:05.128 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:14:05.128 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:14:05.129 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:14:05.129 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:14:05.129 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.133+0000 7fa328d10640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:05.129 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.133+0000 7fa328d10640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:05.132 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.133+0000 7fa328d10640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:14:05.132 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.133+0000 7fa328d10640 1 -- 172.21.15.47:0/2472307793 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa308005080 con 0x7fa3241457f0 2024-03-29T11:14:05.132 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.134+0000 7fa31affd640 1 -- 172.21.15.47:0/2472307793 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7fa31c003ea0 con 0x7fa3241457f0 2024-03-29T11:14:05.133 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.135+0000 7fa31affd640 1 -- 172.21.15.47:0/2472307793 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7fa31c084d40 con 0x7fa3241457f0 2024-03-29T11:14:05.133 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.135+0000 7fa31affd640 10 client.4286.objecter ms_dispatch 0x7fa324000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:14:05.133 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.135+0000 7fa31affd640 3 client.4286.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:14:05.133 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.135+0000 7fa31affd640 3 client.4286.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:14:05.133 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.135+0000 7fa31affd640 20 client.4286.objecter dump_active .. 0 homeless 2024-03-29T11:14:05.133 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.135+0000 7fa32259c640 10 monclient: get_auth_request con 0x7fa30406de90 auth_method 0 2024-03-29T11:14:05.133 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.136+0000 7fa31affd640 10 client.4286.objecter ms_handle_connect 0x7fa30406de90 2024-03-29T11:14:05.134 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.141+0000 7fa31affd640 1 -- 172.21.15.47:0/2472307793 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa31c050620 con 0x7fa3241457f0 2024-03-29T11:14:05.134 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.141+0000 7fa31affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:05.134 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.141+0000 7fa31affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:14:05.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.442+0000 7fa328d10640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:05.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.442+0000 7fa328d10640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:05.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.442+0000 7fa328d10640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:14:05.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.442+0000 7fa328d10640 1 -- 172.21.15.47:0/2472307793 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa308005330 con 0x7fa3241457f0 2024-03-29T11:14:05.421 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.443+0000 7fa31affd640 1 -- 172.21.15.47:0/2472307793 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7fa31c052550 con 0x7fa3241457f0 2024-03-29T11:14:05.421 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.443+0000 7fa31affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:05.421 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.443+0000 7fa31affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:14:05.421 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:14:05.421 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:14:05.425 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.446+0000 7fa328d10640 10 librados: watch_flush enter 2024-03-29T11:14:05.425 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.446+0000 7fa328d10640 10 librados: watch_flush exit 2024-03-29T11:14:05.425 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.446+0000 7fa328d10640 20 client.4286.objecter shutdown clearing up homeless session... 2024-03-29T11:14:05.425 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.446+0000 7fa328d10640 10 client.4286.objecter successfully canceled tick 2024-03-29T11:14:05.425 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.447+0000 7fa328d10640 1 -- 172.21.15.47:0/2472307793 >> v1:172.21.15.121:6816/1964117370 conn(0x7fa30406de90 legacy=0x7fa304070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:05.425 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.447+0000 7fa328d10640 10 monclient: shutdown 2024-03-29T11:14:05.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.447+0000 7fa328d10640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:05.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.447+0000 7fa328d10640 1 -- 172.21.15.47:0/2472307793 >> v1:172.21.15.47:6790/0 conn(0x7fa3241457f0 legacy=0x7fa32413adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:05.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.447+0000 7fa328d10640 1 -- 172.21.15.47:0/2472307793 shutdown_connections 2024-03-29T11:14:05.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.447+0000 7fa328d10640 1 -- 172.21.15.47:0/2472307793 >> 172.21.15.47:0/2472307793 conn(0x7fa324066a00 msgr2=0x7fa3240b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:05.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.447+0000 7fa328d10640 1 -- 172.21.15.47:0/2472307793 shutdown_connections 2024-03-29T11:14:05.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.447+0000 7fa328d10640 1 -- 172.21.15.47:0/2472307793 wait complete. 2024-03-29T11:14:05.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:05.447+0000 7fa328d10640 1 librados: shutdown 2024-03-29T11:14:05.437 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:14:12.243 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:14:12.387 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.408+0000 7f95ad6d4640 10 monclient: get_monmap_and_config 2024-03-29T11:14:12.387 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.408+0000 7f95ad6d4640 10 monclient: build_initial_monmap 2024-03-29T11:14:12.387 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.408+0000 7f95ad6d4640 10 monclient: monmap: 2024-03-29T11:14:12.387 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:14:12.387 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:14:12.387 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:14:12.409697+0000 2024-03-29T11:14:12.387 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:14:12.409697+0000 2024-03-29T11:14:12.387 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:12.387 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:14:12.387 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:14:12.388 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:14:12.388 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:14:12.388 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:14:12.388 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.409+0000 7f95ad6d4640 1 Processor -- start 2024-03-29T11:14:12.388 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.410+0000 7f95ad6d4640 1 -- start start 2024-03-29T11:14:12.388 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.410+0000 7f95ad6d4640 10 monclient: init 2024-03-29T11:14:12.390 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.410+0000 7f95ad6d4640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:12.390 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.410+0000 7f95ad6d4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:14:12.390 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.410+0000 7f95ad6d4640 10 monclient(hunting): picked mon.noname-b con 0x7f95a80743e0 addr v1:172.21.15.47:6790/0 2024-03-29T11:14:12.391 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.410+0000 7f95ad6d4640 10 monclient(hunting): picked mon.noname-c con 0x7f95a80747c0 addr v1:172.21.15.121:6789/0 2024-03-29T11:14:12.391 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.410+0000 7f95ad6d4640 10 monclient(hunting): picked mon.noname-a con 0x7f95a813bd30 addr v1:172.21.15.47:6789/0 2024-03-29T11:14:12.391 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.410+0000 7f95ad6d4640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95a8070f40 con 0x7f95a813bd30 2024-03-29T11:14:12.391 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.410+0000 7f95ad6d4640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95a8071360 con 0x7f95a80747c0 2024-03-29T11:14:12.391 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.410+0000 7f95ad6d4640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95a806f090 con 0x7f95a80743e0 2024-03-29T11:14:12.392 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.410+0000 7f95ad6d4640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:12.392 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.410+0000 7f95ad6d4640 10 monclient(hunting): authenticate will time out at 895.598837s 2024-03-29T11:14:12.392 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.413+0000 7f95a6ffd640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f95a80743e0 0x7f95a806e980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:40091/0 (socket says 172.21.15.47:40091) 2024-03-29T11:14:12.392 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.413+0000 7f95a6ffd640 1 -- 172.21.15.47:0/147244573 learned_addr learned my addr 172.21.15.47:0/147244573 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:14:12.392 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.414+0000 7f95a5ffb640 1 -- 172.21.15.47:0/147244573 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1292163798 0 0) 0x7f95a806f090 con 0x7f95a80743e0 2024-03-29T11:14:12.392 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.414+0000 7f95a5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:14:12.397 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.414+0000 7f95a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:12.397 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.414+0000 7f95a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:12.397 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.414+0000 7f95a5ffb640 10 monclient(hunting): my global_id is 4289 2024-03-29T11:14:12.397 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.414+0000 7f95a5ffb640 1 -- 172.21.15.47:0/147244573 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9594003520 con 0x7f95a80743e0 2024-03-29T11:14:12.397 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.414+0000 7f95a5ffb640 1 -- 172.21.15.47:0/147244573 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4290390676 0 0) 0x7f9594003520 con 0x7f95a80743e0 2024-03-29T11:14:12.397 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.414+0000 7f95a5ffb640 1 -- 172.21.15.47:0/147244573 >> v1:172.21.15.121:6789/0 conn(0x7f95a80747c0 legacy=0x7f95a813b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:14:12.397 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.414+0000 7f95a5ffb640 1 -- 172.21.15.47:0/147244573 >> v1:172.21.15.47:6789/0 conn(0x7f95a813bd30 legacy=0x7f95a813e7b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:14:12.397 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.417+0000 7f95a5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:14:12.397 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.417+0000 7f95a5ffb640 1 monclient: found mon.noname-b 2024-03-29T11:14:12.397 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.417+0000 7f95a5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:12.397 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.417+0000 7f95a5ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-29T11:14:12.397 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.417+0000 7f95a5ffb640 1 -- 172.21.15.47:0/147244573 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95a8072e60 con 0x7f95a80743e0 2024-03-29T11:14:12.398 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.417+0000 7f95a5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:14:12.398 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.417+0000 7f95a5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:14:12.398 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.417+0000 7f95a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:12.398 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.417+0000 7f95a5ffb640 1 -- 172.21.15.47:0/147244573 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f9598001a10 con 0x7f95a80743e0 2024-03-29T11:14:12.398 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.417+0000 7f95a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:12.398 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.417+0000 7f95a5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:14:12.398 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.417+0000 7f95a5ffb640 10 monclient: dump: 2024-03-29T11:14:12.398 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:14:12.398 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:14:12.398 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:12.398 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:12.398 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:12.398 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:14:12.398 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:14:12.399 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:14:12.399 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:14:12.399 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:14:12.399 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.417+0000 7f95ad6d4640 5 monclient: authenticate success, global_id 4289 2024-03-29T11:14:12.399 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.417+0000 7f95ad6d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:12.399 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.418+0000 7f95a5ffb640 1 -- 172.21.15.47:0/147244573 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9598001ca0 con 0x7f95a80743e0 2024-03-29T11:14:12.399 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.418+0000 7f95a5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:12.399 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.418+0000 7f95ad6d4640 10 monclient: get_monmap_and_config success 2024-03-29T11:14:12.400 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.418+0000 7f95ad6d4640 4 set_mon_vals no callback set 2024-03-29T11:14:12.400 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.418+0000 7f95ad6d4640 10 monclient: shutdown 2024-03-29T11:14:12.400 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.418+0000 7f95ad6d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:12.400 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.418+0000 7f95ad6d4640 1 -- 172.21.15.47:0/147244573 >> v1:172.21.15.47:6790/0 conn(0x7f95a80743e0 legacy=0x7f95a806e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:12.400 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.419+0000 7f95ad6d4640 1 -- 172.21.15.47:0/147244573 shutdown_connections 2024-03-29T11:14:12.400 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.419+0000 7f95ad6d4640 1 -- 172.21.15.47:0/147244573 >> 172.21.15.47:0/147244573 conn(0x7f95a80b3f40 msgr2=0x7f95a80b4320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:12.400 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.419+0000 7f95ad6d4640 1 -- 172.21.15.47:0/147244573 shutdown_connections 2024-03-29T11:14:12.402 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.419+0000 7f95ad6d4640 1 -- 172.21.15.47:0/147244573 wait complete. 2024-03-29T11:14:12.402 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.420+0000 7f95ad6d4640 10 monclient: build_initial_monmap 2024-03-29T11:14:12.402 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.420+0000 7f95ad6d4640 10 monclient: monmap: 2024-03-29T11:14:12.402 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:14:12.402 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:14:12.402 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:14:12.402 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:14:12.402 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:12.402 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:14:12.402 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:14:12.402 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:14:12.402 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:14:12.402 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:14:12.403 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.420+0000 7f95ad6d4640 1 librados: starting msgr at 2024-03-29T11:14:12.403 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.420+0000 7f95ad6d4640 1 librados: starting objecter 2024-03-29T11:14:12.403 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.420+0000 7f95ad6d4640 1 Processor -- start 2024-03-29T11:14:12.403 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.421+0000 7f95ad6d4640 1 -- start start 2024-03-29T11:14:12.403 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.421+0000 7f95ad6d4640 1 librados: setting wanted keys 2024-03-29T11:14:12.403 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.421+0000 7f95ad6d4640 1 librados: calling monclient init 2024-03-29T11:14:12.403 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.421+0000 7f95ad6d4640 10 monclient: init 2024-03-29T11:14:12.403 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.421+0000 7f95ad6d4640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:12.403 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.421+0000 7f95ad6d4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:14:12.403 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.421+0000 7f95ad6d4640 10 monclient(hunting): picked mon.noname-b con 0x7f95a80743e0 addr v1:172.21.15.121:6789/0 2024-03-29T11:14:12.403 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.422+0000 7f95ad6d4640 10 monclient(hunting): picked mon.noname-c con 0x7f95a80747c0 addr v1:172.21.15.47:6790/0 2024-03-29T11:14:12.410 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.422+0000 7f95a6ffd640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f95a80743e0 0x7f95a813aa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.47:40680/0 (socket says 172.21.15.47:40680) 2024-03-29T11:14:12.410 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.424+0000 7f95ad6d4640 10 monclient(hunting): picked mon.noname-a con 0x7f95a813bd30 addr v1:172.21.15.47:6789/0 2024-03-29T11:14:12.410 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.424+0000 7f95ad6d4640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95a81e78c0 con 0x7f95a813bd30 2024-03-29T11:14:12.410 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.424+0000 7f95ad6d4640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95a8151d60 con 0x7f95a80743e0 2024-03-29T11:14:12.410 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.424+0000 7f95ad6d4640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95a81e8460 con 0x7f95a80747c0 2024-03-29T11:14:12.410 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.424+0000 7f95ad6d4640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:12.410 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.424+0000 7f95ad6d4640 10 monclient(hunting): authenticate will time out at 895.612324s 2024-03-29T11:14:12.410 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.424+0000 7f95a6ffd640 1 -- 172.21.15.47:0/2262314139 learned_addr learned my addr 172.21.15.47:0/2262314139 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:14:12.411 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.424+0000 7f959f7fe640 10 client.?.objecter ms_handle_connect 0x7f95a80743e0 2024-03-29T11:14:12.411 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.424+0000 7f959f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:12.411 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.424+0000 7f959f7fe640 1 -- 172.21.15.47:0/2262314139 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3142563760 0 0) 0x7f95a8151d60 con 0x7f95a80743e0 2024-03-29T11:14:12.411 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.424+0000 7f959f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:14:12.411 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.424+0000 7f959f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:12.411 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.424+0000 7f959f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:12.411 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.424+0000 7f959f7fe640 10 monclient(hunting): my global_id is 4276 2024-03-29T11:14:12.411 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.424+0000 7f959f7fe640 1 -- 172.21.15.47:0/2262314139 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9588004eb0 con 0x7f95a80743e0 2024-03-29T11:14:12.411 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.424+0000 7f959f7fe640 10 client.?.objecter ms_handle_connect 0x7f95a80747c0 2024-03-29T11:14:12.411 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.424+0000 7f959f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:12.411 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.425+0000 7f959f7fe640 1 -- 172.21.15.47:0/2262314139 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 224981442 0 0) 0x7f9588004eb0 con 0x7f95a80743e0 2024-03-29T11:14:12.411 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.425+0000 7f959f7fe640 1 -- 172.21.15.47:0/2262314139 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95a8151d60 con 0x7f95a80743e0 2024-03-29T11:14:12.411 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.425+0000 7f959f7fe640 1 -- 172.21.15.47:0/2262314139 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f9598004d40 con 0x7f95a80743e0 2024-03-29T11:14:12.411 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.425+0000 7f959f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:12.412 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.425+0000 7f959f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:14:12.412 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.425+0000 7f959f7fe640 10 monclient(hunting): dump: 2024-03-29T11:14:12.412 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:14:12.412 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:14:12.412 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:12.412 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:12.412 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:12.412 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:14:12.412 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:14:12.412 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:14:12.412 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:14:12.412 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:14:12.412 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.425+0000 7f959f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:14:12.412 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.425+0000 7f959f7fe640 1 -- 172.21.15.47:0/2262314139 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 209831563 0 0) 0x7f95a81e8460 con 0x7f95a80747c0 2024-03-29T11:14:12.413 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.425+0000 7f959f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:14:12.413 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.425+0000 7f959f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:12.413 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.425+0000 7f959f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:12.413 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.425+0000 7f959f7fe640 10 monclient(hunting): my global_id is 4292 2024-03-29T11:14:12.413 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.425+0000 7f959f7fe640 1 -- 172.21.15.47:0/2262314139 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9588004eb0 con 0x7f95a80747c0 2024-03-29T11:14:12.413 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.425+0000 7f959f7fe640 1 -- 172.21.15.47:0/2262314139 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3052533649 0 0) 0x7f95a8151d60 con 0x7f95a80743e0 2024-03-29T11:14:12.413 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.425+0000 7f959f7fe640 1 -- 172.21.15.47:0/2262314139 >> v1:172.21.15.47:6790/0 conn(0x7f95a80747c0 legacy=0x7f95a81e7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:12.413 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.425+0000 7f959f7fe640 1 -- 172.21.15.47:0/2262314139 >> v1:172.21.15.47:6789/0 conn(0x7f95a813bd30 legacy=0x7f95a81eeb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:12.413 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.425+0000 7f959f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:14:12.413 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.425+0000 7f959f7fe640 1 monclient: found mon.b 2024-03-29T11:14:12.413 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.425+0000 7f959f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:12.413 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.425+0000 7f959f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:14:12.413 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.425+0000 7f959f7fe640 1 -- 172.21.15.47:0/2262314139 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95a8151fd0 con 0x7f95a80743e0 2024-03-29T11:14:12.413 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.425+0000 7f959f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:14:12.413 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.425+0000 7f959f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:14:12.414 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.425+0000 7f959f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:12.414 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.425+0000 7f95ad6d4640 5 monclient: authenticate success, global_id 4276 2024-03-29T11:14:12.414 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.425+0000 7f95ad6d4640 10 monclient: _renew_subs 2024-03-29T11:14:12.414 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.425+0000 7f95ad6d4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:14:12.414 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.425+0000 7f95ad6d4640 1 -- 172.21.15.47:0/2262314139 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95a80a9680 con 0x7f95a80743e0 2024-03-29T11:14:12.414 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.425+0000 7f95ad6d4640 10 client.4276.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:14:12.414 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.425+0000 7f95ad6d4640 10 monclient: _renew_subs 2024-03-29T11:14:12.414 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.425+0000 7f95ad6d4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:14:12.414 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.425+0000 7f95ad6d4640 1 -- 172.21.15.47:0/2262314139 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f95a814bc30 con 0x7f95a80743e0 2024-03-29T11:14:12.414 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.425+0000 7f95ad6d4640 1 librados: init done 2024-03-29T11:14:12.414 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.426+0000 7f959f7fe640 1 -- 172.21.15.47:0/2262314139 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9598001e90 con 0x7f95a80743e0 2024-03-29T11:14:12.414 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.426+0000 7f959f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:12.414 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.426+0000 7f959f7fe640 1 -- 172.21.15.47:0/2262314139 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f95980052f0 con 0x7f95a80743e0 2024-03-29T11:14:12.414 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.426+0000 7f959f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:12.414 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.426+0000 7f959f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:14:12.415 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.426+0000 7f959f7fe640 10 monclient: dump: 2024-03-29T11:14:12.415 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:14:12.415 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:14:12.415 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:12.415 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:12.415 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:12.415 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:14:12.415 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:14:12.415 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:14:12.415 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:14:12.415 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:14:12.415 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.427+0000 7f95ad6d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:12.415 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.427+0000 7f95ad6d4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:12.415 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.427+0000 7f95ad6d4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:14:12.416 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.427+0000 7f95ad6d4640 1 -- 172.21.15.47:0/2262314139 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95a8070190 con 0x7f95a80743e0 2024-03-29T11:14:12.416 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.428+0000 7f959f7fe640 1 -- 172.21.15.47:0/2262314139 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7f9598005450 con 0x7f95a80743e0 2024-03-29T11:14:12.416 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.428+0000 7f95a57fa640 4 set_mon_vals no callback set 2024-03-29T11:14:12.416 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.429+0000 7f959f7fe640 1 -- 172.21.15.47:0/2262314139 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7f95980850d0 con 0x7f95a80743e0 2024-03-29T11:14:12.416 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.429+0000 7f959f7fe640 10 client.4276.objecter ms_dispatch 0x7f95a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:14:12.416 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.429+0000 7f959f7fe640 3 client.4276.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:14:12.416 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.429+0000 7f959f7fe640 3 client.4276.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:14:12.416 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.429+0000 7f959f7fe640 20 client.4276.objecter dump_active .. 0 homeless 2024-03-29T11:14:12.416 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.429+0000 7f95a67fc640 10 monclient: get_auth_request con 0x7f958806de00 auth_method 0 2024-03-29T11:14:12.416 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.430+0000 7f959f7fe640 10 client.4276.objecter ms_handle_connect 0x7f958806de00 2024-03-29T11:14:12.417 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.436+0000 7f959f7fe640 1 -- 172.21.15.47:0/2262314139 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f95980509b0 con 0x7f95a80743e0 2024-03-29T11:14:12.417 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.436+0000 7f959f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:12.417 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.436+0000 7f959f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:14:12.761 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.783+0000 7f95ad6d4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:12.761 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.783+0000 7f95ad6d4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:12.761 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.783+0000 7f95ad6d4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:14:12.761 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.783+0000 7f95ad6d4640 1 -- 172.21.15.47:0/2262314139 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f95a8075f10 con 0x7f95a80743e0 2024-03-29T11:14:12.762 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.785+0000 7f959f7fe640 1 -- 172.21.15.47:0/2262314139 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7f95980528e0 con 0x7f95a80743e0 2024-03-29T11:14:12.762 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.785+0000 7f959f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:12.762 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.785+0000 7f959f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:14:12.763 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:14:12.763 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:14:12.766 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.788+0000 7f95ad6d4640 10 librados: watch_flush enter 2024-03-29T11:14:12.767 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.789+0000 7f95ad6d4640 10 librados: watch_flush exit 2024-03-29T11:14:12.767 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.789+0000 7f95ad6d4640 20 client.4276.objecter shutdown clearing up homeless session... 2024-03-29T11:14:12.767 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.789+0000 7f95ad6d4640 10 client.4276.objecter successfully canceled tick 2024-03-29T11:14:12.767 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.789+0000 7f95ad6d4640 1 -- 172.21.15.47:0/2262314139 >> v1:172.21.15.121:6816/1964117370 conn(0x7f958806de00 legacy=0x7f95880702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:12.767 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.789+0000 7f95ad6d4640 10 monclient: shutdown 2024-03-29T11:14:12.767 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.789+0000 7f95ad6d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:12.767 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.789+0000 7f95ad6d4640 1 -- 172.21.15.47:0/2262314139 >> v1:172.21.15.121:6789/0 conn(0x7f95a80743e0 legacy=0x7f95a813aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:12.767 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.789+0000 7f95ad6d4640 1 -- 172.21.15.47:0/2262314139 shutdown_connections 2024-03-29T11:14:12.767 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.789+0000 7f95ad6d4640 1 -- 172.21.15.47:0/2262314139 >> 172.21.15.47:0/2262314139 conn(0x7f95a80b3f40 msgr2=0x7f95a806e980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:12.768 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.790+0000 7f95ad6d4640 1 -- 172.21.15.47:0/2262314139 shutdown_connections 2024-03-29T11:14:12.768 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.790+0000 7f95ad6d4640 1 -- 172.21.15.47:0/2262314139 wait complete. 2024-03-29T11:14:12.768 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:12.790+0000 7f95ad6d4640 1 librados: shutdown 2024-03-29T11:14:12.785 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:14:19.593 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:14:19.739 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.760+0000 7ff435d82640 10 monclient: get_monmap_and_config 2024-03-29T11:14:19.739 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.760+0000 7ff435d82640 10 monclient: build_initial_monmap 2024-03-29T11:14:19.739 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.760+0000 7ff435d82640 10 monclient: monmap: 2024-03-29T11:14:19.739 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:14:19.739 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:14:19.739 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:14:19.762795+0000 2024-03-29T11:14:19.740 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:14:19.762795+0000 2024-03-29T11:14:19.740 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:19.740 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:14:19.740 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:14:19.740 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:14:19.740 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:14:19.740 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:14:19.742 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.762+0000 7ff435d82640 1 Processor -- start 2024-03-29T11:14:19.742 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.762+0000 7ff435d82640 1 -- start start 2024-03-29T11:14:19.742 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.762+0000 7ff435d82640 10 monclient: init 2024-03-29T11:14:19.742 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.763+0000 7ff435d82640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:19.742 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.763+0000 7ff435d82640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:14:19.742 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.763+0000 7ff435d82640 10 monclient(hunting): picked mon.noname-a con 0x7ff4300b1f60 addr v1:172.21.15.47:6789/0 2024-03-29T11:14:19.742 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.763+0000 7ff435d82640 10 monclient(hunting): picked mon.noname-c con 0x7ff430071eb0 addr v1:172.21.15.121:6789/0 2024-03-29T11:14:19.742 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.763+0000 7ff435d82640 10 monclient(hunting): picked mon.noname-b con 0x7ff430143d10 addr v1:172.21.15.47:6790/0 2024-03-29T11:14:19.742 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.763+0000 7ff435d82640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff43006ef80 con 0x7ff4300b1f60 2024-03-29T11:14:19.742 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.763+0000 7ff435d82640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff43006f260 con 0x7ff430071eb0 2024-03-29T11:14:19.743 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.763+0000 7ff435d82640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4300b40c0 con 0x7ff430143d10 2024-03-29T11:14:19.743 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.763+0000 7ff435d82640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:19.743 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.763+0000 7ff435d82640 10 monclient(hunting): authenticate will time out at 902.952802s 2024-03-29T11:14:19.744 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.766+0000 7ff42f7fe640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7ff4300b1f60 0x7ff430143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:52113/0 (socket says 172.21.15.47:52113) 2024-03-29T11:14:19.744 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.766+0000 7ff42f7fe640 1 -- 172.21.15.47:0/3738671440 learned_addr learned my addr 172.21.15.47:0/3738671440 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:14:19.746 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.768+0000 7ff42e7fc640 1 -- 172.21.15.47:0/3738671440 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3594385663 0 0) 0x7ff43006ef80 con 0x7ff4300b1f60 2024-03-29T11:14:19.747 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.768+0000 7ff42e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:14:19.747 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.768+0000 7ff42e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:19.747 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.768+0000 7ff42e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:19.747 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.768+0000 7ff42e7fc640 10 monclient(hunting): my global_id is 4281 2024-03-29T11:14:19.747 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.768+0000 7ff42e7fc640 1 -- 172.21.15.47:0/3738671440 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff418003520 con 0x7ff4300b1f60 2024-03-29T11:14:19.747 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.768+0000 7ff42e7fc640 1 -- 172.21.15.47:0/3738671440 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4101553520 0 0) 0x7ff418003520 con 0x7ff4300b1f60 2024-03-29T11:14:19.747 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.768+0000 7ff42e7fc640 1 -- 172.21.15.47:0/3738671440 >> v1:172.21.15.47:6790/0 conn(0x7ff430143d10 legacy=0x7ff430146b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:14:19.748 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.768+0000 7ff42e7fc640 1 -- 172.21.15.47:0/3738671440 >> v1:172.21.15.121:6789/0 conn(0x7ff430071eb0 legacy=0x7ff430146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:19.748 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.768+0000 7ff42e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:14:19.748 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.769+0000 7ff42e7fc640 1 monclient: found mon.noname-a 2024-03-29T11:14:19.749 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.769+0000 7ff42e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:19.749 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.769+0000 7ff42e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-29T11:14:19.749 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.769+0000 7ff42e7fc640 1 -- 172.21.15.47:0/3738671440 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff430155d70 con 0x7ff4300b1f60 2024-03-29T11:14:19.749 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.769+0000 7ff42e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:14:19.749 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.769+0000 7ff42e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:14:19.749 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.769+0000 7ff42e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:19.749 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.769+0000 7ff42e7fc640 1 -- 172.21.15.47:0/3738671440 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7ff424001690 con 0x7ff4300b1f60 2024-03-29T11:14:19.749 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.769+0000 7ff42e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:19.749 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.769+0000 7ff42e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:14:19.749 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.769+0000 7ff42e7fc640 10 monclient: dump: 2024-03-29T11:14:19.749 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:14:19.749 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:14:19.749 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:19.749 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:19.749 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:19.750 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:14:19.750 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:14:19.750 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:14:19.750 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:14:19.750 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:14:19.750 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.769+0000 7ff435d82640 5 monclient: authenticate success, global_id 4281 2024-03-29T11:14:19.750 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.769+0000 7ff435d82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:19.750 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.769+0000 7ff42e7fc640 1 -- 172.21.15.47:0/3738671440 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff424001860 con 0x7ff4300b1f60 2024-03-29T11:14:19.750 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.769+0000 7ff42e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:19.750 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.769+0000 7ff435d82640 10 monclient: get_monmap_and_config success 2024-03-29T11:14:19.750 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.769+0000 7ff435d82640 4 set_mon_vals no callback set 2024-03-29T11:14:19.750 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.770+0000 7ff435d82640 10 monclient: shutdown 2024-03-29T11:14:19.750 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.770+0000 7ff435d82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:19.750 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.770+0000 7ff435d82640 1 -- 172.21.15.47:0/3738671440 >> v1:172.21.15.47:6789/0 conn(0x7ff4300b1f60 legacy=0x7ff430143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:19.753 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.771+0000 7ff435d82640 1 -- 172.21.15.47:0/3738671440 shutdown_connections 2024-03-29T11:14:19.753 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.771+0000 7ff435d82640 1 -- 172.21.15.47:0/3738671440 >> 172.21.15.47:0/3738671440 conn(0x7ff430066a00 msgr2=0x7ff430066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:19.753 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.772+0000 7ff435d82640 1 -- 172.21.15.47:0/3738671440 shutdown_connections 2024-03-29T11:14:19.753 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.772+0000 7ff435d82640 1 -- 172.21.15.47:0/3738671440 wait complete. 2024-03-29T11:14:19.753 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.773+0000 7ff435d82640 10 monclient: build_initial_monmap 2024-03-29T11:14:19.753 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.773+0000 7ff435d82640 10 monclient: monmap: 2024-03-29T11:14:19.754 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:14:19.754 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:14:19.754 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:14:19.754 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:14:19.754 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:19.754 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:14:19.754 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:14:19.754 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:14:19.754 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:14:19.754 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:14:19.754 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.773+0000 7ff435d82640 1 librados: starting msgr at 2024-03-29T11:14:19.754 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.773+0000 7ff435d82640 1 librados: starting objecter 2024-03-29T11:14:19.754 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.773+0000 7ff435d82640 1 Processor -- start 2024-03-29T11:14:19.755 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.773+0000 7ff435d82640 1 -- start start 2024-03-29T11:14:19.755 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.773+0000 7ff435d82640 1 librados: setting wanted keys 2024-03-29T11:14:19.755 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.773+0000 7ff435d82640 1 librados: calling monclient init 2024-03-29T11:14:19.755 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.773+0000 7ff435d82640 10 monclient: init 2024-03-29T11:14:19.755 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.774+0000 7ff435d82640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:19.755 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.774+0000 7ff435d82640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:14:19.755 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.774+0000 7ff435d82640 10 monclient(hunting): picked mon.noname-c con 0x7ff4300b1f60 addr v1:172.21.15.47:6790/0 2024-03-29T11:14:19.755 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.774+0000 7ff435d82640 10 monclient(hunting): picked mon.noname-a con 0x7ff430143d10 addr v1:172.21.15.47:6789/0 2024-03-29T11:14:19.755 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.774+0000 7ff435d82640 10 monclient(hunting): picked mon.noname-b con 0x7ff4301399c0 addr v1:172.21.15.121:6789/0 2024-03-29T11:14:19.755 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.774+0000 7ff435d82640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff43013ac20 con 0x7ff430143d10 2024-03-29T11:14:19.755 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.774+0000 7ff435d82640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff43013bad0 con 0x7ff4301399c0 2024-03-29T11:14:19.755 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.774+0000 7ff435d82640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff43013a870 con 0x7ff4300b1f60 2024-03-29T11:14:19.756 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.774+0000 7ff435d82640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:19.756 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.774+0000 7ff435d82640 10 monclient(hunting): authenticate will time out at 902.963658s 2024-03-29T11:14:19.758 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.776+0000 7ff42ffff640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7ff4301399c0 0x7ff4300b76c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.47:36862/0 (socket says 172.21.15.47:36862) 2024-03-29T11:14:19.758 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.776+0000 7ff42ffff640 1 -- 172.21.15.47:0/3202667332 learned_addr learned my addr 172.21.15.47:0/3202667332 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:14:19.758 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.777+0000 7ff417fff640 10 client.?.objecter ms_handle_connect 0x7ff430143d10 2024-03-29T11:14:19.758 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.777+0000 7ff417fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:19.758 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.777+0000 7ff417fff640 1 -- 172.21.15.47:0/3202667332 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 735315308 0 0) 0x7ff43006f260 con 0x7ff430143d10 2024-03-29T11:14:19.758 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.777+0000 7ff417fff640 10 monclient(hunting): _negotiate 2024-03-29T11:14:19.758 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.777+0000 7ff417fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:19.758 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.777+0000 7ff417fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:19.758 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.777+0000 7ff417fff640 10 monclient(hunting): my global_id is 4284 2024-03-29T11:14:19.758 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.777+0000 7ff417fff640 1 -- 172.21.15.47:0/3202667332 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff41c004eb0 con 0x7ff430143d10 2024-03-29T11:14:19.759 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.777+0000 7ff417fff640 1 -- 172.21.15.47:0/3202667332 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 336536278 0 0) 0x7ff41c004eb0 con 0x7ff430143d10 2024-03-29T11:14:19.759 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.777+0000 7ff417fff640 1 -- 172.21.15.47:0/3202667332 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff43006f260 con 0x7ff430143d10 2024-03-29T11:14:19.759 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.777+0000 7ff417fff640 1 -- 172.21.15.47:0/3202667332 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7ff4200016c0 con 0x7ff430143d10 2024-03-29T11:14:19.759 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.777+0000 7ff417fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:19.759 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.777+0000 7ff417fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:14:19.759 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.777+0000 7ff417fff640 10 monclient(hunting): dump: 2024-03-29T11:14:19.759 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:14:19.759 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:14:19.759 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:19.759 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:19.759 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:19.760 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:14:19.760 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:14:19.760 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:14:19.760 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:14:19.760 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:14:19.760 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.777+0000 7ff417fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:14:19.760 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.778+0000 7ff417fff640 10 client.?.objecter ms_handle_connect 0x7ff4300b1f60 2024-03-29T11:14:19.760 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.778+0000 7ff417fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:19.760 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.778+0000 7ff417fff640 10 client.?.objecter ms_handle_connect 0x7ff4301399c0 2024-03-29T11:14:19.761 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.778+0000 7ff417fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:19.761 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.778+0000 7ff417fff640 1 -- 172.21.15.47:0/3202667332 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 768840504 0 0) 0x7ff43013a870 con 0x7ff4300b1f60 2024-03-29T11:14:19.761 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.778+0000 7ff417fff640 10 monclient(hunting): _negotiate 2024-03-29T11:14:19.761 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.778+0000 7ff417fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:19.761 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.778+0000 7ff417fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:19.761 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.778+0000 7ff417fff640 10 monclient(hunting): my global_id is 4295 2024-03-29T11:14:19.762 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.778+0000 7ff417fff640 1 -- 172.21.15.47:0/3202667332 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff41c004eb0 con 0x7ff4300b1f60 2024-03-29T11:14:19.764 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.780+0000 7ff417fff640 1 -- 172.21.15.47:0/3202667332 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3618652122 0 0) 0x7ff43013bad0 con 0x7ff4301399c0 2024-03-29T11:14:19.765 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.780+0000 7ff417fff640 10 monclient(hunting): _negotiate 2024-03-29T11:14:19.765 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.780+0000 7ff417fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:19.765 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.780+0000 7ff417fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:19.765 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.780+0000 7ff417fff640 10 monclient(hunting): my global_id is 4282 2024-03-29T11:14:19.765 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.780+0000 7ff417fff640 1 -- 172.21.15.47:0/3202667332 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff43013a870 con 0x7ff4301399c0 2024-03-29T11:14:19.765 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.780+0000 7ff417fff640 1 -- 172.21.15.47:0/3202667332 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2107142600 0 0) 0x7ff41c004eb0 con 0x7ff4300b1f60 2024-03-29T11:14:19.766 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.780+0000 7ff417fff640 1 -- 172.21.15.47:0/3202667332 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff43013bad0 con 0x7ff4300b1f60 2024-03-29T11:14:19.766 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.780+0000 7ff417fff640 1 -- 172.21.15.47:0/3202667332 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1644640664 0 0) 0x7ff43006f260 con 0x7ff430143d10 2024-03-29T11:14:19.766 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.780+0000 7ff417fff640 1 -- 172.21.15.47:0/3202667332 >> v1:172.21.15.47:6790/0 conn(0x7ff4300b1f60 legacy=0x7ff43013a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:19.766 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.780+0000 7ff417fff640 1 -- 172.21.15.47:0/3202667332 >> v1:172.21.15.121:6789/0 conn(0x7ff4301399c0 legacy=0x7ff4300b76c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:19.766 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.780+0000 7ff417fff640 10 monclient: _finish_hunting 0 2024-03-29T11:14:19.766 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.780+0000 7ff417fff640 1 monclient: found mon.a 2024-03-29T11:14:19.766 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.780+0000 7ff417fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:19.766 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.780+0000 7ff417fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:14:19.767 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.780+0000 7ff417fff640 1 -- 172.21.15.47:0/3202667332 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4301f26b0 con 0x7ff430143d10 2024-03-29T11:14:19.767 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.780+0000 7ff417fff640 10 monclient: _finish_auth 0 2024-03-29T11:14:19.767 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.780+0000 7ff417fff640 10 monclient: _check_auth_tickets 2024-03-29T11:14:19.767 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.780+0000 7ff417fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:19.767 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.781+0000 7ff417fff640 1 -- 172.21.15.47:0/3202667332 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff420001cf0 con 0x7ff430143d10 2024-03-29T11:14:19.767 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.781+0000 7ff417fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:19.767 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.781+0000 7ff42d7fa640 4 set_mon_vals no callback set 2024-03-29T11:14:19.767 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.781+0000 7ff417fff640 1 -- 172.21.15.47:0/3202667332 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7ff4200037b0 con 0x7ff430143d10 2024-03-29T11:14:19.767 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.781+0000 7ff417fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:19.767 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.781+0000 7ff417fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:14:19.767 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.781+0000 7ff417fff640 10 monclient: dump: 2024-03-29T11:14:19.767 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:14:19.767 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:14:19.767 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:19.768 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:19.768 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:19.768 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:14:19.768 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:14:19.768 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:14:19.768 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:14:19.768 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:14:19.768 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.781+0000 7ff435d82640 5 monclient: authenticate success, global_id 4284 2024-03-29T11:14:19.768 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.781+0000 7ff435d82640 10 monclient: _renew_subs 2024-03-29T11:14:19.768 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.781+0000 7ff435d82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:14:19.768 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.781+0000 7ff435d82640 1 -- 172.21.15.47:0/3202667332 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4301ef680 con 0x7ff430143d10 2024-03-29T11:14:19.768 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.781+0000 7ff435d82640 10 client.4284.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:14:19.768 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.781+0000 7ff435d82640 10 monclient: _renew_subs 2024-03-29T11:14:19.768 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.781+0000 7ff435d82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:14:19.768 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.781+0000 7ff435d82640 1 -- 172.21.15.47:0/3202667332 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4301efb60 con 0x7ff430143d10 2024-03-29T11:14:19.769 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.781+0000 7ff435d82640 1 librados: init done 2024-03-29T11:14:19.769 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.782+0000 7ff435d82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:19.769 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.782+0000 7ff435d82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:19.769 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.782+0000 7ff435d82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:14:19.769 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.782+0000 7ff435d82640 1 -- 172.21.15.47:0/3202667332 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff43013aa00 con 0x7ff430143d10 2024-03-29T11:14:19.769 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.783+0000 7ff417fff640 1 -- 172.21.15.47:0/3202667332 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7ff420019780 con 0x7ff430143d10 2024-03-29T11:14:19.769 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.784+0000 7ff417fff640 1 -- 172.21.15.47:0/3202667332 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7ff4200836e0 con 0x7ff430143d10 2024-03-29T11:14:19.769 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.784+0000 7ff417fff640 10 client.4284.objecter ms_dispatch 0x7ff430000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:14:19.769 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.784+0000 7ff417fff640 3 client.4284.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:14:19.769 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.784+0000 7ff417fff640 3 client.4284.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:14:19.769 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.784+0000 7ff417fff640 20 client.4284.objecter dump_active .. 0 homeless 2024-03-29T11:14:19.769 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.784+0000 7ff42f7fe640 10 monclient: get_auth_request con 0x7ff41c06e070 auth_method 0 2024-03-29T11:14:19.769 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.785+0000 7ff417fff640 10 client.4284.objecter ms_handle_connect 0x7ff41c06e070 2024-03-29T11:14:19.773 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.794+0000 7ff417fff640 1 -- 172.21.15.47:0/3202667332 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff42004efc0 con 0x7ff430143d10 2024-03-29T11:14:19.773 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.794+0000 7ff417fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:19.773 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:19.794+0000 7ff417fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:14:20.100 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:20.122+0000 7ff435d82640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:20.100 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:20.122+0000 7ff435d82640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:20.101 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:20.122+0000 7ff435d82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:14:20.101 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:20.122+0000 7ff435d82640 1 -- 172.21.15.47:0/3202667332 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff430075f10 con 0x7ff430143d10 2024-03-29T11:14:20.101 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:20.123+0000 7ff417fff640 1 -- 172.21.15.47:0/3202667332 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7ff420050ef0 con 0x7ff430143d10 2024-03-29T11:14:20.101 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:20.123+0000 7ff417fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:20.101 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:20.123+0000 7ff417fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:14:20.102 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:14:20.102 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:14:20.107 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:20.128+0000 7ff435d82640 10 librados: watch_flush enter 2024-03-29T11:14:20.107 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:20.128+0000 7ff435d82640 10 librados: watch_flush exit 2024-03-29T11:14:20.107 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:20.128+0000 7ff435d82640 20 client.4284.objecter shutdown clearing up homeless session... 2024-03-29T11:14:20.107 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:20.128+0000 7ff435d82640 10 client.4284.objecter successfully canceled tick 2024-03-29T11:14:20.107 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:20.128+0000 7ff435d82640 1 -- 172.21.15.47:0/3202667332 >> v1:172.21.15.121:6816/1964117370 conn(0x7ff41c06e070 legacy=0x7ff41c070530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:20.107 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:20.128+0000 7ff435d82640 10 monclient: shutdown 2024-03-29T11:14:20.107 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:20.128+0000 7ff435d82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:20.107 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:20.128+0000 7ff435d82640 1 -- 172.21.15.47:0/3202667332 >> v1:172.21.15.47:6789/0 conn(0x7ff430143d10 legacy=0x7ff4300b6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:20.107 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:20.128+0000 7ff435d82640 1 -- 172.21.15.47:0/3202667332 shutdown_connections 2024-03-29T11:14:20.107 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:20.128+0000 7ff435d82640 1 -- 172.21.15.47:0/3202667332 >> 172.21.15.47:0/3202667332 conn(0x7ff430066a00 msgr2=0x7ff430066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:20.108 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:20.128+0000 7ff435d82640 1 -- 172.21.15.47:0/3202667332 shutdown_connections 2024-03-29T11:14:20.108 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:20.129+0000 7ff435d82640 1 -- 172.21.15.47:0/3202667332 wait complete. 2024-03-29T11:14:20.108 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:20.129+0000 7ff435d82640 1 librados: shutdown 2024-03-29T11:14:20.126 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:14:26.933 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:14:27.091 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.112+0000 7f5ecffff640 10 monclient: get_monmap_and_config 2024-03-29T11:14:27.092 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.112+0000 7f5ecffff640 10 monclient: build_initial_monmap 2024-03-29T11:14:27.092 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.112+0000 7f5ecffff640 10 monclient: monmap: 2024-03-29T11:14:27.092 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:14:27.092 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:14:27.092 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:14:27.113295+0000 2024-03-29T11:14:27.092 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:14:27.113295+0000 2024-03-29T11:14:27.092 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:27.093 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:14:27.093 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:14:27.093 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:14:27.093 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:14:27.093 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:14:27.094 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.115+0000 7f5ecffff640 1 Processor -- start 2024-03-29T11:14:27.095 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.115+0000 7f5ecffff640 1 -- start start 2024-03-29T11:14:27.095 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.115+0000 7f5ecffff640 10 monclient: init 2024-03-29T11:14:27.095 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.116+0000 7f5ecffff640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:27.095 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.116+0000 7f5ecffff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:14:27.095 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.116+0000 7f5ecffff640 10 monclient(hunting): picked mon.noname-a con 0x7f5ec802cc50 addr v1:172.21.15.47:6789/0 2024-03-29T11:14:27.095 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.116+0000 7f5ecffff640 10 monclient(hunting): picked mon.noname-b con 0x7f5ec802d3a0 addr v1:172.21.15.47:6790/0 2024-03-29T11:14:27.095 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.116+0000 7f5ecffff640 10 monclient(hunting): picked mon.noname-c con 0x7f5ec80c6290 addr v1:172.21.15.121:6789/0 2024-03-29T11:14:27.099 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.116+0000 7f5ecffff640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ec80c56f0 con 0x7f5ec802cc50 2024-03-29T11:14:27.099 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.117+0000 7f5ecffff640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ec802e080 con 0x7f5ec80c6290 2024-03-29T11:14:27.099 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.117+0000 7f5ecffff640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ec802e360 con 0x7f5ec802d3a0 2024-03-29T11:14:27.099 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.117+0000 7f5ecffff640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:27.099 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.117+0000 7f5ecffff640 10 monclient(hunting): authenticate will time out at 910.305054s 2024-03-29T11:14:27.099 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.119+0000 7f5eceffd640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f5ec802cc50 0x7f5ec80c2ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:58029/0 (socket says 172.21.15.47:58029) 2024-03-29T11:14:27.099 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.119+0000 7f5eceffd640 1 -- 172.21.15.47:0/2892223699 learned_addr learned my addr 172.21.15.47:0/2892223699 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:14:27.106 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.123+0000 7f5ecdffb640 1 -- 172.21.15.47:0/2892223699 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1604787123 0 0) 0x7f5ec80c56f0 con 0x7f5ec802cc50 2024-03-29T11:14:27.106 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.123+0000 7f5ecdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:14:27.107 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.123+0000 7f5ecdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:27.108 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.123+0000 7f5ecdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:27.108 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.123+0000 7f5ecdffb640 10 monclient(hunting): my global_id is 4287 2024-03-29T11:14:27.108 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.124+0000 7f5ecdffb640 1 -- 172.21.15.47:0/2892223699 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5eb8003520 con 0x7f5ec802cc50 2024-03-29T11:14:27.108 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.124+0000 7f5ecdffb640 1 -- 172.21.15.47:0/2892223699 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3546317893 0 0) 0x7f5eb8003520 con 0x7f5ec802cc50 2024-03-29T11:14:27.108 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.124+0000 7f5ecdffb640 1 -- 172.21.15.47:0/2892223699 >> v1:172.21.15.47:6790/0 conn(0x7f5ec802d3a0 legacy=0x7f5ec80c5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:27.108 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.124+0000 7f5ecdffb640 1 -- 172.21.15.47:0/2892223699 >> v1:172.21.15.121:6789/0 conn(0x7f5ec80c6290 legacy=0x7f5ec80d0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:27.108 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.124+0000 7f5ecdffb640 10 monclient: _finish_hunting 0 2024-03-29T11:14:27.109 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.124+0000 7f5ecdffb640 1 monclient: found mon.noname-a 2024-03-29T11:14:27.109 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.124+0000 7f5ecdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:27.109 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.124+0000 7f5ecdffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-29T11:14:27.109 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.124+0000 7f5ecdffb640 1 -- 172.21.15.47:0/2892223699 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ec80c8c10 con 0x7f5ec802cc50 2024-03-29T11:14:27.109 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.125+0000 7f5ecdffb640 10 monclient: _finish_auth 0 2024-03-29T11:14:27.109 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.125+0000 7f5ecdffb640 10 monclient: _check_auth_tickets 2024-03-29T11:14:27.110 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.125+0000 7f5ecdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:27.110 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.125+0000 7f5ecffff640 5 monclient: authenticate success, global_id 4287 2024-03-29T11:14:27.110 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.125+0000 7f5ecffff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:27.110 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.125+0000 7f5ecdffb640 1 -- 172.21.15.47:0/2892223699 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f5ed004f730 con 0x7f5ec802cc50 2024-03-29T11:14:27.110 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.125+0000 7f5ecdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:27.110 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.125+0000 7f5ecdffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:14:27.110 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.125+0000 7f5ecdffb640 10 monclient: dump: 2024-03-29T11:14:27.110 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:14:27.110 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:14:27.113 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:27.113 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:27.113 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:27.114 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:14:27.115 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:14:27.115 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:14:27.115 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:14:27.115 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:14:27.115 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.125+0000 7f5ecffff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:27.117 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.125+0000 7f5ecdffb640 1 -- 172.21.15.47:0/2892223699 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ed00008d0 con 0x7f5ec802cc50 2024-03-29T11:14:27.117 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.125+0000 7f5ecdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:27.117 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.125+0000 7f5ecffff640 10 monclient: get_monmap_and_config success 2024-03-29T11:14:27.117 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.125+0000 7f5ecffff640 4 set_mon_vals no callback set 2024-03-29T11:14:27.117 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.126+0000 7f5ecdffb640 1 -- 172.21.15.47:0/2892223699 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f5ed00758b0 con 0x7f5ec802cc50 2024-03-29T11:14:27.117 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.126+0000 7f5ecdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:27.117 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.126+0000 7f5ecdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:14:27.118 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.126+0000 7f5ecdffb640 10 monclient: dump: 2024-03-29T11:14:27.118 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:14:27.118 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:14:27.118 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:27.118 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:27.119 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:27.119 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:14:27.119 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:14:27.119 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:14:27.119 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:14:27.120 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:14:27.120 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.126+0000 7f5ecffff640 10 monclient: shutdown 2024-03-29T11:14:27.120 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.126+0000 7f5ecffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:27.120 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.126+0000 7f5ecffff640 1 -- 172.21.15.47:0/2892223699 >> v1:172.21.15.47:6789/0 conn(0x7f5ec802cc50 legacy=0x7f5ec80c2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:27.128 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.129+0000 7f5ecffff640 1 -- 172.21.15.47:0/2892223699 shutdown_connections 2024-03-29T11:14:27.129 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.129+0000 7f5ecffff640 1 -- 172.21.15.47:0/2892223699 >> 172.21.15.47:0/2892223699 conn(0x7f5ec80058f0 msgr2=0x7f5ec8005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:27.129 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.130+0000 7f5ecffff640 1 -- 172.21.15.47:0/2892223699 shutdown_connections 2024-03-29T11:14:27.129 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.130+0000 7f5ecffff640 1 -- 172.21.15.47:0/2892223699 wait complete. 2024-03-29T11:14:27.130 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.131+0000 7f5ecffff640 10 monclient: build_initial_monmap 2024-03-29T11:14:27.130 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.131+0000 7f5ecffff640 10 monclient: monmap: 2024-03-29T11:14:27.130 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:14:27.130 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:14:27.130 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:14:27.130 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:14:27.130 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:27.130 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:14:27.130 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:14:27.130 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:14:27.131 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:14:27.131 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:14:27.131 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.131+0000 7f5ecffff640 1 librados: starting msgr at 2024-03-29T11:14:27.131 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.131+0000 7f5ecffff640 1 librados: starting objecter 2024-03-29T11:14:27.131 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.132+0000 7f5ecffff640 1 Processor -- start 2024-03-29T11:14:27.132 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.132+0000 7f5ecffff640 1 -- start start 2024-03-29T11:14:27.132 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.132+0000 7f5ecffff640 1 librados: setting wanted keys 2024-03-29T11:14:27.132 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.132+0000 7f5ecffff640 1 librados: calling monclient init 2024-03-29T11:14:27.132 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.132+0000 7f5ecffff640 10 monclient: init 2024-03-29T11:14:27.132 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.133+0000 7f5ecffff640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:27.132 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.133+0000 7f5ecffff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:14:27.132 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.133+0000 7f5ecffff640 10 monclient(hunting): picked mon.noname-b con 0x7f5ec802cc50 addr v1:172.21.15.121:6789/0 2024-03-29T11:14:27.133 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.133+0000 7f5ecffff640 10 monclient(hunting): picked mon.noname-a con 0x7f5ec802d3a0 addr v1:172.21.15.47:6789/0 2024-03-29T11:14:27.133 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.133+0000 7f5ecffff640 10 monclient(hunting): picked mon.noname-c con 0x7f5ec80c6290 addr v1:172.21.15.47:6790/0 2024-03-29T11:14:27.133 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.133+0000 7f5ecffff640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ec816dd40 con 0x7f5ec802d3a0 2024-03-29T11:14:27.134 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.133+0000 7f5ecffff640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ec80bb7b0 con 0x7f5ec802cc50 2024-03-29T11:14:27.134 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.133+0000 7f5ecffff640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ec80bba90 con 0x7f5ec80c6290 2024-03-29T11:14:27.135 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.133+0000 7f5ecffff640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:27.135 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.133+0000 7f5ecffff640 10 monclient(hunting): authenticate will time out at 910.321138s 2024-03-29T11:14:27.135 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.137+0000 7f5ece7fc640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f5ec802d3a0 0x7f5ec816cf20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:58267/0 (socket says 172.21.15.47:58267) 2024-03-29T11:14:27.135 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.137+0000 7f5ece7fc640 1 -- 172.21.15.47:0/3551440908 learned_addr learned my addr 172.21.15.47:0/3551440908 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:14:27.135 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.137+0000 7f5eb77fe640 10 client.?.objecter ms_handle_connect 0x7f5ec802d3a0 2024-03-29T11:14:27.135 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.138+0000 7f5eb77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:27.135 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.138+0000 7f5eb77fe640 1 -- 172.21.15.47:0/3551440908 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2110850772 0 0) 0x7f5ec816dd40 con 0x7f5ec802d3a0 2024-03-29T11:14:27.135 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.138+0000 7f5eb77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:14:27.135 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.138+0000 7f5eb77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:27.136 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.138+0000 7f5eb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:27.136 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.138+0000 7f5eb77fe640 10 monclient(hunting): my global_id is 4290 2024-03-29T11:14:27.136 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.138+0000 7f5eb77fe640 1 -- 172.21.15.47:0/3551440908 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ebc004eb0 con 0x7f5ec802d3a0 2024-03-29T11:14:27.136 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.138+0000 7f5eb77fe640 1 -- 172.21.15.47:0/3551440908 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2344070055 0 0) 0x7f5ebc004eb0 con 0x7f5ec802d3a0 2024-03-29T11:14:27.136 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.138+0000 7f5eb77fe640 1 -- 172.21.15.47:0/3551440908 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ec816dd40 con 0x7f5ec802d3a0 2024-03-29T11:14:27.136 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.138+0000 7f5eb77fe640 1 -- 172.21.15.47:0/3551440908 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f5ec4001990 con 0x7f5ec802d3a0 2024-03-29T11:14:27.136 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.138+0000 7f5eb77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:27.136 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.138+0000 7f5eb77fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:14:27.136 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.138+0000 7f5eb77fe640 10 monclient(hunting): dump: 2024-03-29T11:14:27.136 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:14:27.136 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:14:27.137 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:27.137 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:27.137 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:27.137 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:14:27.137 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:14:27.137 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:14:27.137 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:14:27.137 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:14:27.137 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.138+0000 7f5eb77fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:14:27.137 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.139+0000 7f5eb77fe640 10 client.?.objecter ms_handle_connect 0x7f5ec802cc50 2024-03-29T11:14:27.138 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.139+0000 7f5eb77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:27.138 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.139+0000 7f5eb77fe640 1 -- 172.21.15.47:0/3551440908 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 503108733 0 0) 0x7f5ec80bb7b0 con 0x7f5ec802cc50 2024-03-29T11:14:27.138 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.139+0000 7f5eb77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:14:27.138 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.139+0000 7f5eb77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:27.138 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.139+0000 7f5eb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:27.138 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.139+0000 7f5eb77fe640 10 monclient(hunting): my global_id is 4288 2024-03-29T11:14:27.138 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.139+0000 7f5eb77fe640 1 -- 172.21.15.47:0/3551440908 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ebc004eb0 con 0x7f5ec802cc50 2024-03-29T11:14:27.138 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.139+0000 7f5eb77fe640 1 -- 172.21.15.47:0/3551440908 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2843442261 0 0) 0x7f5ec816dd40 con 0x7f5ec802d3a0 2024-03-29T11:14:27.139 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.139+0000 7f5eb77fe640 1 -- 172.21.15.47:0/3551440908 >> v1:172.21.15.47:6790/0 conn(0x7f5ec80c6290 legacy=0x7f5ec816d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:27.139 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.139+0000 7f5eb77fe640 1 -- 172.21.15.47:0/3551440908 >> v1:172.21.15.121:6789/0 conn(0x7f5ec802cc50 legacy=0x7f5ec80bc470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:27.139 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.139+0000 7f5eb77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:14:27.139 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.139+0000 7f5eb77fe640 1 monclient: found mon.a 2024-03-29T11:14:27.140 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.139+0000 7f5eb77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:27.140 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.139+0000 7f5eb77fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:14:27.141 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.139+0000 7f5eb77fe640 1 -- 172.21.15.47:0/3551440908 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ec80bbd70 con 0x7f5ec802d3a0 2024-03-29T11:14:27.141 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.139+0000 7f5eb77fe640 10 monclient: _finish_auth 0 2024-03-29T11:14:27.141 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.139+0000 7f5eb77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:14:27.141 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.139+0000 7f5eb77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:27.141 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.140+0000 7f5ecffff640 5 monclient: authenticate success, global_id 4290 2024-03-29T11:14:27.141 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.140+0000 7f5ecffff640 10 monclient: _renew_subs 2024-03-29T11:14:27.141 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.140+0000 7f5ecffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:14:27.141 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.140+0000 7f5ecffff640 1 -- 172.21.15.47:0/3551440908 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ec80bc050 con 0x7f5ec802d3a0 2024-03-29T11:14:27.141 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.140+0000 7f5ecffff640 10 client.4290.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:14:27.141 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.140+0000 7f5ecffff640 10 monclient: _renew_subs 2024-03-29T11:14:27.141 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.140+0000 7f5ecffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:14:27.142 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.140+0000 7f5ecffff640 1 -- 172.21.15.47:0/3551440908 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ec80ba320 con 0x7f5ec802d3a0 2024-03-29T11:14:27.142 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.140+0000 7f5ecffff640 1 librados: init done 2024-03-29T11:14:27.142 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.142+0000 7f5ecffff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:27.142 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.142+0000 7f5ecffff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:27.142 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.142+0000 7f5ecffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:14:27.143 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.142+0000 7f5ecffff640 1 -- 172.21.15.47:0/3551440908 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ec8004cd0 con 0x7f5ec802d3a0 2024-03-29T11:14:27.143 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.144+0000 7f5eb77fe640 1 -- 172.21.15.47:0/3551440908 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ec4001c90 con 0x7f5ec802d3a0 2024-03-29T11:14:27.143 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.144+0000 7f5eb77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:27.143 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.144+0000 7f5eb77fe640 1 -- 172.21.15.47:0/3551440908 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f5ec4003a60 con 0x7f5ec802d3a0 2024-03-29T11:14:27.143 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.144+0000 7f5eb77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:27.143 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.144+0000 7f5eb77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:14:27.143 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.144+0000 7f5eb77fe640 10 monclient: dump: 2024-03-29T11:14:27.143 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:14:27.143 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:14:27.143 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:27.143 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:27.143 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:27.144 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:14:27.144 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:14:27.144 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:14:27.144 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:14:27.144 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:14:27.144 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.144+0000 7f5ecd7fa640 4 set_mon_vals no callback set 2024-03-29T11:14:27.145 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.145+0000 7f5eb77fe640 1 -- 172.21.15.47:0/3551440908 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7f5ec4003c80 con 0x7f5ec802d3a0 2024-03-29T11:14:27.145 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.148+0000 7f5eb77fe640 1 -- 172.21.15.47:0/3551440908 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7f5ec4084ad0 con 0x7f5ec802d3a0 2024-03-29T11:14:27.145 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.148+0000 7f5eb77fe640 10 client.4290.objecter ms_dispatch 0x7f5ed0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:14:27.145 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.149+0000 7f5eb77fe640 3 client.4290.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:14:27.145 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.149+0000 7f5eceffd640 10 monclient: get_auth_request con 0x7f5ebc06dc80 auth_method 0 2024-03-29T11:14:27.145 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.149+0000 7f5eb77fe640 3 client.4290.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:14:27.146 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.149+0000 7f5eb77fe640 20 client.4290.objecter dump_active .. 0 homeless 2024-03-29T11:14:27.146 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.149+0000 7f5eb77fe640 10 client.4290.objecter ms_handle_connect 0x7f5ebc06dc80 2024-03-29T11:14:27.147 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.152+0000 7f5eb77fe640 1 -- 172.21.15.47:0/3551440908 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5ec40503b0 con 0x7f5ec802d3a0 2024-03-29T11:14:27.147 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.152+0000 7f5eb77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:27.147 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.152+0000 7f5eb77fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:14:27.465 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.488+0000 7f5ecffff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:27.466 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.488+0000 7f5ecffff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:27.466 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.488+0000 7f5ecffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:14:27.466 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.488+0000 7f5ecffff640 1 -- 172.21.15.47:0/3551440908 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5ec80327f0 con 0x7f5ec802d3a0 2024-03-29T11:14:27.471 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:14:27.472 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:14:27.472 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.491+0000 7f5eb77fe640 1 -- 172.21.15.47:0/3551440908 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7f5ec40522e0 con 0x7f5ec802d3a0 2024-03-29T11:14:27.472 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.491+0000 7f5eb77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:27.472 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.491+0000 7f5eb77fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:14:27.475 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.497+0000 7f5ecffff640 10 librados: watch_flush enter 2024-03-29T11:14:27.475 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.497+0000 7f5ecffff640 10 librados: watch_flush exit 2024-03-29T11:14:27.475 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.497+0000 7f5ecffff640 20 client.4290.objecter shutdown clearing up homeless session... 2024-03-29T11:14:27.475 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.497+0000 7f5ecffff640 10 client.4290.objecter successfully canceled tick 2024-03-29T11:14:27.475 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.497+0000 7f5ecffff640 1 -- 172.21.15.47:0/3551440908 >> v1:172.21.15.121:6816/1964117370 conn(0x7f5ebc06dc80 legacy=0x7f5ebc070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:27.475 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.497+0000 7f5ecffff640 10 monclient: shutdown 2024-03-29T11:14:27.475 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.497+0000 7f5ecffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:27.475 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.497+0000 7f5ecffff640 1 -- 172.21.15.47:0/3551440908 >> v1:172.21.15.47:6789/0 conn(0x7f5ec802d3a0 legacy=0x7f5ec816cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:27.476 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.497+0000 7f5ecffff640 1 -- 172.21.15.47:0/3551440908 shutdown_connections 2024-03-29T11:14:27.476 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.497+0000 7f5ecffff640 1 -- 172.21.15.47:0/3551440908 >> 172.21.15.47:0/3551440908 conn(0x7f5ec80058f0 msgr2=0x7f5ec80043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:27.479 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.501+0000 7f5ecffff640 1 -- 172.21.15.47:0/3551440908 shutdown_connections 2024-03-29T11:14:27.479 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.501+0000 7f5ecffff640 1 -- 172.21.15.47:0/3551440908 wait complete. 2024-03-29T11:14:27.479 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:27.501+0000 7f5ecffff640 1 librados: shutdown 2024-03-29T11:14:27.500 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:14:34.310 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:14:34.462 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.484+0000 7f2dca6c0640 10 monclient: get_monmap_and_config 2024-03-29T11:14:34.462 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.484+0000 7f2dca6c0640 10 monclient: build_initial_monmap 2024-03-29T11:14:34.462 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.484+0000 7f2dca6c0640 10 monclient: monmap: 2024-03-29T11:14:34.462 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:14:34.462 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:14:34.462 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:14:34.485348+0000 2024-03-29T11:14:34.462 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:14:34.485348+0000 2024-03-29T11:14:34.463 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:34.463 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:14:34.463 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:14:34.463 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:14:34.463 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:14:34.463 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:14:34.463 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.485+0000 7f2dca6c0640 1 Processor -- start 2024-03-29T11:14:34.464 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.485+0000 7f2dca6c0640 1 -- start start 2024-03-29T11:14:34.464 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.485+0000 7f2dca6c0640 10 monclient: init 2024-03-29T11:14:34.465 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.487+0000 7f2dca6c0640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:34.465 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.487+0000 7f2dca6c0640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:14:34.465 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.487+0000 7f2dca6c0640 10 monclient(hunting): picked mon.noname-a con 0x7f2dc40b0860 addr v1:172.21.15.47:6789/0 2024-03-29T11:14:34.465 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.487+0000 7f2dca6c0640 10 monclient(hunting): picked mon.noname-c con 0x7f2dc413ef30 addr v1:172.21.15.121:6789/0 2024-03-29T11:14:34.465 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.487+0000 7f2dca6c0640 10 monclient(hunting): picked mon.noname-b con 0x7f2dc40af8e0 addr v1:172.21.15.47:6790/0 2024-03-29T11:14:34.466 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.487+0000 7f2dca6c0640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dc40b0c40 con 0x7f2dc40b0860 2024-03-29T11:14:34.466 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.487+0000 7f2dca6c0640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dc40aed10 con 0x7f2dc413ef30 2024-03-29T11:14:34.466 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.487+0000 7f2dca6c0640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dc40afd50 con 0x7f2dc40af8e0 2024-03-29T11:14:34.466 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.487+0000 7f2dca6c0640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:34.466 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.487+0000 7f2dca6c0640 10 monclient(hunting): authenticate will time out at 917.675602s 2024-03-29T11:14:34.470 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.493+0000 7f2dc3fff640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f2dc40b0860 0x7f2dc413e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:43845/0 (socket says 172.21.15.47:43845) 2024-03-29T11:14:34.470 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.493+0000 7f2dc3fff640 1 -- 172.21.15.47:0/1865128305 learned_addr learned my addr 172.21.15.47:0/1865128305 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:14:34.471 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.493+0000 7f2dc2ffd640 1 -- 172.21.15.47:0/1865128305 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2738874 0 0) 0x7f2dc40aed10 con 0x7f2dc413ef30 2024-03-29T11:14:34.472 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.493+0000 7f2dc2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:14:34.472 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.493+0000 7f2dc2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:34.472 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.493+0000 7f2dc2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:34.472 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.493+0000 7f2dc2ffd640 10 monclient(hunting): my global_id is 4291 2024-03-29T11:14:34.472 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.493+0000 7f2dc2ffd640 1 -- 172.21.15.47:0/1865128305 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dac003520 con 0x7f2dc413ef30 2024-03-29T11:14:34.472 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.494+0000 7f2dc2ffd640 1 -- 172.21.15.47:0/1865128305 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 424761541 0 0) 0x7f2dc40b0c40 con 0x7f2dc40b0860 2024-03-29T11:14:34.472 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.494+0000 7f2dc2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:14:34.472 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.494+0000 7f2dc2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:34.472 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.494+0000 7f2dc2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:34.472 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.494+0000 7f2dc2ffd640 10 monclient(hunting): my global_id is 4293 2024-03-29T11:14:34.473 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.494+0000 7f2dc2ffd640 1 -- 172.21.15.47:0/1865128305 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dac004a20 con 0x7f2dc40b0860 2024-03-29T11:14:34.474 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.494+0000 7f2dc2ffd640 1 -- 172.21.15.47:0/1865128305 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3083002628 0 0) 0x7f2dc40afd50 con 0x7f2dc40af8e0 2024-03-29T11:14:34.475 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.494+0000 7f2dc2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:14:34.475 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.494+0000 7f2dc2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:34.475 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.494+0000 7f2dc2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:34.475 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.494+0000 7f2dc2ffd640 10 monclient(hunting): my global_id is 4301 2024-03-29T11:14:34.475 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.494+0000 7f2dc2ffd640 1 -- 172.21.15.47:0/1865128305 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dc40b0c40 con 0x7f2dc40af8e0 2024-03-29T11:14:34.475 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.496+0000 7f2dc2ffd640 1 -- 172.21.15.47:0/1865128305 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3695713424 0 0) 0x7f2dc40b0c40 con 0x7f2dc40af8e0 2024-03-29T11:14:34.475 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.496+0000 7f2dc2ffd640 1 -- 172.21.15.47:0/1865128305 >> v1:172.21.15.121:6789/0 conn(0x7f2dc413ef30 legacy=0x7f2dc40af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:34.475 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.496+0000 7f2dc2ffd640 1 -- 172.21.15.47:0/1865128305 >> v1:172.21.15.47:6789/0 conn(0x7f2dc40b0860 legacy=0x7f2dc413e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:34.475 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.496+0000 7f2dc2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:14:34.475 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.496+0000 7f2dc2ffd640 1 monclient: found mon.noname-b 2024-03-29T11:14:34.475 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.496+0000 7f2dc2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:34.476 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.496+0000 7f2dc2ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-29T11:14:34.476 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.496+0000 7f2dc2ffd640 1 -- 172.21.15.47:0/1865128305 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dc40ab1b0 con 0x7f2dc40af8e0 2024-03-29T11:14:34.476 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.496+0000 7f2dc2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:14:34.476 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.496+0000 7f2dc2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:14:34.476 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.496+0000 7f2dc2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:34.476 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.496+0000 7f2dc2ffd640 1 -- 172.21.15.47:0/1865128305 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f2db80015b0 con 0x7f2dc40af8e0 2024-03-29T11:14:34.476 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.497+0000 7f2dc2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:34.476 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.497+0000 7f2dc2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:14:34.476 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.497+0000 7f2dc2ffd640 10 monclient: dump: 2024-03-29T11:14:34.476 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:14:34.476 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:14:34.476 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:34.477 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:34.477 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:34.477 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:14:34.477 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:14:34.477 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:14:34.477 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:14:34.477 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:14:34.477 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.497+0000 7f2dca6c0640 5 monclient: authenticate success, global_id 4301 2024-03-29T11:14:34.477 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.497+0000 7f2dca6c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:34.479 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.500+0000 7f2dc2ffd640 1 -- 172.21.15.47:0/1865128305 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2db8001710 con 0x7f2dc40af8e0 2024-03-29T11:14:34.479 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.500+0000 7f2dc2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:34.479 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.500+0000 7f2dc2ffd640 1 -- 172.21.15.47:0/1865128305 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f2db8001b60 con 0x7f2dc40af8e0 2024-03-29T11:14:34.479 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.500+0000 7f2dc2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:34.479 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.500+0000 7f2dc2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:14:34.479 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.500+0000 7f2dc2ffd640 10 monclient: dump: 2024-03-29T11:14:34.479 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:14:34.479 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:14:34.479 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:34.480 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:34.480 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:34.480 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:14:34.480 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:14:34.480 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:14:34.480 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:14:34.480 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:14:34.480 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.500+0000 7f2dca6c0640 10 monclient: get_monmap_and_config success 2024-03-29T11:14:34.480 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.500+0000 7f2dca6c0640 4 set_mon_vals no callback set 2024-03-29T11:14:34.480 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.501+0000 7f2dca6c0640 10 monclient: shutdown 2024-03-29T11:14:34.480 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.501+0000 7f2dca6c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:34.480 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.501+0000 7f2dca6c0640 1 -- 172.21.15.47:0/1865128305 >> v1:172.21.15.47:6790/0 conn(0x7f2dc40af8e0 legacy=0x7f2dc4149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:34.481 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.502+0000 7f2dca6c0640 1 -- 172.21.15.47:0/1865128305 shutdown_connections 2024-03-29T11:14:34.481 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.502+0000 7f2dca6c0640 1 -- 172.21.15.47:0/1865128305 >> 172.21.15.47:0/1865128305 conn(0x7f2dc406b660 msgr2=0x7f2dc406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:34.482 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.505+0000 7f2dca6c0640 1 -- 172.21.15.47:0/1865128305 shutdown_connections 2024-03-29T11:14:34.486 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.507+0000 7f2dca6c0640 1 -- 172.21.15.47:0/1865128305 wait complete. 2024-03-29T11:14:34.486 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.509+0000 7f2dca6c0640 10 monclient: build_initial_monmap 2024-03-29T11:14:34.486 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.509+0000 7f2dca6c0640 10 monclient: monmap: 2024-03-29T11:14:34.486 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:14:34.486 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:14:34.487 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:14:34.487 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:14:34.487 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:34.487 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:14:34.487 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:14:34.487 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:14:34.487 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:14:34.487 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:14:34.488 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.509+0000 7f2dca6c0640 1 librados: starting msgr at 2024-03-29T11:14:34.489 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.509+0000 7f2dca6c0640 1 librados: starting objecter 2024-03-29T11:14:34.489 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.509+0000 7f2dca6c0640 1 Processor -- start 2024-03-29T11:14:34.489 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.510+0000 7f2dca6c0640 1 -- start start 2024-03-29T11:14:34.489 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.510+0000 7f2dca6c0640 1 librados: setting wanted keys 2024-03-29T11:14:34.489 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.510+0000 7f2dca6c0640 1 librados: calling monclient init 2024-03-29T11:14:34.489 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.510+0000 7f2dca6c0640 10 monclient: init 2024-03-29T11:14:34.489 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.511+0000 7f2dca6c0640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:34.489 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.511+0000 7f2dca6c0640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:14:34.489 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.511+0000 7f2dca6c0640 10 monclient(hunting): picked mon.noname-c con 0x7f2dc40af8e0 addr v1:172.21.15.47:6790/0 2024-03-29T11:14:34.492 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.514+0000 7f2dca6c0640 10 monclient(hunting): picked mon.noname-b con 0x7f2dc40b0860 addr v1:172.21.15.121:6789/0 2024-03-29T11:14:34.493 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.514+0000 7f2dca6c0640 10 monclient(hunting): picked mon.noname-a con 0x7f2dc413ef30 addr v1:172.21.15.47:6789/0 2024-03-29T11:14:34.493 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.514+0000 7f2dca6c0640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dc4137970 con 0x7f2dc413ef30 2024-03-29T11:14:34.493 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.514+0000 7f2dca6c0640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dc41370f0 con 0x7f2dc40b0860 2024-03-29T11:14:34.493 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.514+0000 7f2dca6c0640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dc4137bc0 con 0x7f2dc40af8e0 2024-03-29T11:14:34.493 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.514+0000 7f2dca6c0640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:34.493 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.514+0000 7f2dca6c0640 10 monclient(hunting): authenticate will time out at 917.702897s 2024-03-29T11:14:34.493 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.515+0000 7f2dc37fe640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f2dc40b0860 0x7f2dc41d3320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.47:59355/0 (socket says 172.21.15.47:59355) 2024-03-29T11:14:34.493 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.515+0000 7f2dc37fe640 1 -- 172.21.15.47:0/2697511539 learned_addr learned my addr 172.21.15.47:0/2697511539 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:14:34.496 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.517+0000 7f2dc0ff9640 10 client.?.objecter ms_handle_connect 0x7f2dc40b0860 2024-03-29T11:14:34.496 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.517+0000 7f2dc0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:34.496 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.517+0000 7f2dc0ff9640 1 -- 172.21.15.47:0/2697511539 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2192590883 0 0) 0x7f2dc41370f0 con 0x7f2dc40b0860 2024-03-29T11:14:34.496 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.517+0000 7f2dc0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:14:34.496 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.517+0000 7f2dc0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:34.496 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.517+0000 7f2dc0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:34.496 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.517+0000 7f2dc0ff9640 10 monclient(hunting): my global_id is 4294 2024-03-29T11:14:34.496 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.517+0000 7f2dc0ff9640 1 -- 172.21.15.47:0/2697511539 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2db0004eb0 con 0x7f2dc40b0860 2024-03-29T11:14:34.496 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.517+0000 7f2dc0ff9640 1 -- 172.21.15.47:0/2697511539 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1680623850 0 0) 0x7f2db0004eb0 con 0x7f2dc40b0860 2024-03-29T11:14:34.497 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.518+0000 7f2dc0ff9640 1 -- 172.21.15.47:0/2697511539 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2dc41370f0 con 0x7f2dc40b0860 2024-03-29T11:14:34.497 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.518+0000 7f2dc0ff9640 1 -- 172.21.15.47:0/2697511539 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f2db4001940 con 0x7f2dc40b0860 2024-03-29T11:14:34.497 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.518+0000 7f2dc0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:34.497 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.518+0000 7f2dc0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:14:34.497 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.518+0000 7f2dc0ff9640 10 monclient(hunting): dump: 2024-03-29T11:14:34.497 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:14:34.497 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:14:34.497 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:34.497 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:34.497 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:34.497 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:14:34.497 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:14:34.497 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:14:34.497 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:14:34.497 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:14:34.498 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.518+0000 7f2dc0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:14:34.498 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.518+0000 7f2dc0ff9640 1 -- 172.21.15.47:0/2697511539 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2414515545 0 0) 0x7f2dc41370f0 con 0x7f2dc40b0860 2024-03-29T11:14:34.498 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.518+0000 7f2dc0ff9640 1 -- 172.21.15.47:0/2697511539 >> v1:172.21.15.47:6790/0 conn(0x7f2dc40af8e0 legacy=0x7f2dc4136470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:14:34.500 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.520+0000 7f2dc0ff9640 1 -- 172.21.15.47:0/2697511539 >> v1:172.21.15.47:6789/0 conn(0x7f2dc413ef30 legacy=0x7f2dc41ddaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:34.500 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.520+0000 7f2dc0ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:14:34.500 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.520+0000 7f2dc0ff9640 1 monclient: found mon.b 2024-03-29T11:14:34.500 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.520+0000 7f2dc0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:34.500 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.520+0000 7f2dc0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:14:34.500 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.520+0000 7f2dc0ff9640 1 -- 172.21.15.47:0/2697511539 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dc4137ea0 con 0x7f2dc40b0860 2024-03-29T11:14:34.500 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.520+0000 7f2dc0ff9640 10 monclient: _finish_auth 0 2024-03-29T11:14:34.500 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.520+0000 7f2dc0ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:14:34.500 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.520+0000 7f2dc0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:34.501 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.520+0000 7f2dca6c0640 5 monclient: authenticate success, global_id 4294 2024-03-29T11:14:34.501 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.520+0000 7f2dca6c0640 10 monclient: _renew_subs 2024-03-29T11:14:34.501 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.520+0000 7f2dca6c0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:14:34.501 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.520+0000 7f2dca6c0640 1 -- 172.21.15.47:0/2697511539 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dc4136b80 con 0x7f2dc40b0860 2024-03-29T11:14:34.501 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.520+0000 7f2dca6c0640 10 client.4294.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:14:34.501 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.520+0000 7f2dca6c0640 10 monclient: _renew_subs 2024-03-29T11:14:34.501 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.520+0000 7f2dca6c0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:14:34.501 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.520+0000 7f2dca6c0640 1 -- 172.21.15.47:0/2697511539 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dc4136e30 con 0x7f2dc40b0860 2024-03-29T11:14:34.501 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.521+0000 7f2dca6c0640 1 librados: init done 2024-03-29T11:14:34.502 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.522+0000 7f2dc0ff9640 1 -- 172.21.15.47:0/2697511539 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2db4001aa0 con 0x7f2dc40b0860 2024-03-29T11:14:34.502 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.522+0000 7f2dc0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:34.502 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.522+0000 7f2dc0ff9640 1 -- 172.21.15.47:0/2697511539 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f2db40038a0 con 0x7f2dc40b0860 2024-03-29T11:14:34.502 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.522+0000 7f2dc0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:34.502 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.522+0000 7f2dc0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:14:34.502 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.522+0000 7f2dc0ff9640 10 monclient: dump: 2024-03-29T11:14:34.502 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:14:34.502 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:14:34.502 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:34.503 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:34.503 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:34.503 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:14:34.503 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:14:34.503 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:14:34.503 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:14:34.503 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:14:34.503 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.522+0000 7f2dc27fc640 4 set_mon_vals no callback set 2024-03-29T11:14:34.503 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.523+0000 7f2dca6c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:34.503 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.524+0000 7f2dca6c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:34.503 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.524+0000 7f2dca6c0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:14:34.503 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.524+0000 7f2dca6c0640 1 -- 172.21.15.47:0/2697511539 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2da4005080 con 0x7f2dc40b0860 2024-03-29T11:14:34.504 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.526+0000 7f2dc0ff9640 1 -- 172.21.15.47:0/2697511539 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7f2db4003a00 con 0x7f2dc40b0860 2024-03-29T11:14:34.505 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.526+0000 7f2dc0ff9640 1 -- 172.21.15.47:0/2697511539 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7f2db4084a50 con 0x7f2dc40b0860 2024-03-29T11:14:34.505 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.526+0000 7f2dc0ff9640 10 client.4294.objecter ms_dispatch 0x7f2dc4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:14:34.505 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.526+0000 7f2dc0ff9640 3 client.4294.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:14:34.505 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.526+0000 7f2dc0ff9640 3 client.4294.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:14:34.505 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.527+0000 7f2dc0ff9640 20 client.4294.objecter dump_active .. 0 homeless 2024-03-29T11:14:34.507 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.529+0000 7f2dc3fff640 10 monclient: get_auth_request con 0x7f2db006dc70 auth_method 0 2024-03-29T11:14:34.508 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.530+0000 7f2dc0ff9640 10 client.4294.objecter ms_handle_connect 0x7f2db006dc70 2024-03-29T11:14:34.516 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.539+0000 7f2dc0ff9640 1 -- 172.21.15.47:0/2697511539 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2db4050160 con 0x7f2dc40b0860 2024-03-29T11:14:34.517 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.539+0000 7f2dc0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:34.517 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.539+0000 7f2dc0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:14:34.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.875+0000 7f2dca6c0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:34.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.875+0000 7f2dca6c0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:34.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.875+0000 7f2dca6c0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:14:34.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.875+0000 7f2dca6c0640 1 -- 172.21.15.47:0/2697511539 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2da4005330 con 0x7f2dc40b0860 2024-03-29T11:14:34.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.878+0000 7f2dc0ff9640 1 -- 172.21.15.47:0/2697511539 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7f2db4052260 con 0x7f2dc40b0860 2024-03-29T11:14:34.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.878+0000 7f2dc0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:34.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.878+0000 7f2dc0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:14:34.859 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:14:34.860 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:14:34.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.882+0000 7f2dca6c0640 10 librados: watch_flush enter 2024-03-29T11:14:34.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.882+0000 7f2dca6c0640 10 librados: watch_flush exit 2024-03-29T11:14:34.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.882+0000 7f2dca6c0640 20 client.4294.objecter shutdown clearing up homeless session... 2024-03-29T11:14:34.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.882+0000 7f2dca6c0640 10 client.4294.objecter successfully canceled tick 2024-03-29T11:14:34.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.882+0000 7f2dca6c0640 1 -- 172.21.15.47:0/2697511539 >> v1:172.21.15.121:6816/1964117370 conn(0x7f2db006dc70 legacy=0x7f2db0070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:34.861 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.882+0000 7f2dca6c0640 10 monclient: shutdown 2024-03-29T11:14:34.861 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.882+0000 7f2dca6c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:34.861 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.882+0000 7f2dca6c0640 1 -- 172.21.15.47:0/2697511539 >> v1:172.21.15.121:6789/0 conn(0x7f2dc40b0860 legacy=0x7f2dc41d3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:34.861 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.884+0000 7f2dca6c0640 1 -- 172.21.15.47:0/2697511539 shutdown_connections 2024-03-29T11:14:34.861 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.884+0000 7f2dca6c0640 1 -- 172.21.15.47:0/2697511539 >> 172.21.15.47:0/2697511539 conn(0x7f2dc406b660 msgr2=0x7f2dc413e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:34.862 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.884+0000 7f2dca6c0640 1 -- 172.21.15.47:0/2697511539 shutdown_connections 2024-03-29T11:14:34.862 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.884+0000 7f2dca6c0640 1 -- 172.21.15.47:0/2697511539 wait complete. 2024-03-29T11:14:34.862 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:34.884+0000 7f2dca6c0640 1 librados: shutdown 2024-03-29T11:14:34.882 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:14:41.691 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:14:41.844 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.863+0000 7f308a11c640 10 monclient: get_monmap_and_config 2024-03-29T11:14:41.844 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.863+0000 7f308a11c640 10 monclient: build_initial_monmap 2024-03-29T11:14:41.844 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.864+0000 7f308a11c640 10 monclient: monmap: 2024-03-29T11:14:41.845 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:14:41.845 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:14:41.845 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:14:41.864974+0000 2024-03-29T11:14:41.845 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:14:41.864974+0000 2024-03-29T11:14:41.845 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:41.845 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:14:41.845 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:14:41.846 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:14:41.846 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:14:41.846 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:14:41.847 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.868+0000 7f308a11c640 1 Processor -- start 2024-03-29T11:14:41.847 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.870+0000 7f308a11c640 1 -- start start 2024-03-29T11:14:41.847 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.870+0000 7f308a11c640 10 monclient: init 2024-03-29T11:14:41.848 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.870+0000 7f308a11c640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:41.848 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.870+0000 7f308a11c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:14:41.848 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.871+0000 7f308a11c640 10 monclient(hunting): picked mon.noname-c con 0x7f30840ae250 addr v1:172.21.15.121:6789/0 2024-03-29T11:14:41.849 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.871+0000 7f308a11c640 10 monclient(hunting): picked mon.noname-a con 0x7f308413f970 addr v1:172.21.15.47:6789/0 2024-03-29T11:14:41.849 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.871+0000 7f308a11c640 10 monclient(hunting): picked mon.noname-b con 0x7f3084142800 addr v1:172.21.15.47:6790/0 2024-03-29T11:14:41.849 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.871+0000 7f308a11c640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30840afe00 con 0x7f308413f970 2024-03-29T11:14:41.849 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.871+0000 7f308a11c640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3084142c10 con 0x7f30840ae250 2024-03-29T11:14:41.849 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.871+0000 7f308a11c640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30840afb20 con 0x7f3084142800 2024-03-29T11:14:41.849 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.871+0000 7f308a11c640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:41.849 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.871+0000 7f308a11c640 10 monclient(hunting): authenticate will time out at 925.059677s 2024-03-29T11:14:41.851 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.874+0000 7f30837fe640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f30840ae250 0x7f308413f260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.47:48215/0 (socket says 172.21.15.47:48215) 2024-03-29T11:14:41.851 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.874+0000 7f30837fe640 1 -- 172.21.15.47:0/3476088597 learned_addr learned my addr 172.21.15.47:0/3476088597 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:14:41.852 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.874+0000 7f30827fc640 1 -- 172.21.15.47:0/3476088597 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1423501072 0 0) 0x7f3084142c10 con 0x7f30840ae250 2024-03-29T11:14:41.852 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.874+0000 7f30827fc640 10 monclient(hunting): _negotiate 2024-03-29T11:14:41.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.874+0000 7f30827fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:41.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.874+0000 7f30827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:41.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.874+0000 7f30827fc640 10 monclient(hunting): my global_id is 4297 2024-03-29T11:14:41.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.874+0000 7f30827fc640 1 -- 172.21.15.47:0/3476088597 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f306c003520 con 0x7f30840ae250 2024-03-29T11:14:41.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.875+0000 7f30827fc640 1 -- 172.21.15.47:0/3476088597 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 893730048 0 0) 0x7f306c003520 con 0x7f30840ae250 2024-03-29T11:14:41.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.875+0000 7f30827fc640 1 -- 172.21.15.47:0/3476088597 >> v1:172.21.15.47:6790/0 conn(0x7f3084142800 legacy=0x7f308414da10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:14:41.855 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.877+0000 7f30827fc640 1 -- 172.21.15.47:0/3476088597 >> v1:172.21.15.47:6789/0 conn(0x7f308413f970 legacy=0x7f30841420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:14:41.855 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.877+0000 7f30827fc640 10 monclient: _finish_hunting 0 2024-03-29T11:14:41.855 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.877+0000 7f30827fc640 1 monclient: found mon.noname-c 2024-03-29T11:14:41.855 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.877+0000 7f30827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:41.855 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.877+0000 7f30827fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:14:41.855 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.877+0000 7f30827fc640 1 -- 172.21.15.47:0/3476088597 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30840aab80 con 0x7f30840ae250 2024-03-29T11:14:41.855 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.877+0000 7f30827fc640 10 monclient: _finish_auth 0 2024-03-29T11:14:41.855 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.877+0000 7f30827fc640 10 monclient: _check_auth_tickets 2024-03-29T11:14:41.855 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.877+0000 7f30827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:41.855 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.877+0000 7f30827fc640 1 -- 172.21.15.47:0/3476088597 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f307c001a10 con 0x7f30840ae250 2024-03-29T11:14:41.855 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.877+0000 7f30827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:41.855 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.877+0000 7f30827fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:14:41.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.877+0000 7f30827fc640 10 monclient: dump: 2024-03-29T11:14:41.856 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:14:41.856 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:14:41.856 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:41.856 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:41.856 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:41.856 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:14:41.856 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:14:41.856 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:14:41.856 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:14:41.856 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:14:41.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.877+0000 7f308a11c640 5 monclient: authenticate success, global_id 4297 2024-03-29T11:14:41.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.877+0000 7f308a11c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:41.858 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.877+0000 7f30827fc640 1 -- 172.21.15.47:0/3476088597 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f307c001be0 con 0x7f30840ae250 2024-03-29T11:14:41.858 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.878+0000 7f30827fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:41.859 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.878+0000 7f30827fc640 1 -- 172.21.15.47:0/3476088597 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f307c002030 con 0x7f30840ae250 2024-03-29T11:14:41.859 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.878+0000 7f30827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:41.859 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.878+0000 7f30827fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:14:41.859 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.878+0000 7f30827fc640 10 monclient: dump: 2024-03-29T11:14:41.859 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:14:41.859 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:14:41.859 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:41.859 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:41.859 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:41.859 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:14:41.859 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:14:41.859 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:14:41.859 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:14:41.860 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:14:41.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.878+0000 7f308a11c640 10 monclient: get_monmap_and_config success 2024-03-29T11:14:41.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.878+0000 7f308a11c640 4 set_mon_vals no callback set 2024-03-29T11:14:41.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.878+0000 7f308a11c640 10 monclient: shutdown 2024-03-29T11:14:41.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.878+0000 7f308a11c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:41.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.878+0000 7f308a11c640 1 -- 172.21.15.47:0/3476088597 >> v1:172.21.15.121:6789/0 conn(0x7f30840ae250 legacy=0x7f308413f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:41.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.879+0000 7f308a11c640 1 -- 172.21.15.47:0/3476088597 shutdown_connections 2024-03-29T11:14:41.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.879+0000 7f308a11c640 1 -- 172.21.15.47:0/3476088597 >> 172.21.15.47:0/3476088597 conn(0x7f30840b35e0 msgr2=0x7f30840b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:41.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.879+0000 7f308a11c640 1 -- 172.21.15.47:0/3476088597 shutdown_connections 2024-03-29T11:14:41.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.879+0000 7f308a11c640 1 -- 172.21.15.47:0/3476088597 wait complete. 2024-03-29T11:14:41.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.880+0000 7f308a11c640 10 monclient: build_initial_monmap 2024-03-29T11:14:41.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.880+0000 7f308a11c640 10 monclient: monmap: 2024-03-29T11:14:41.860 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:14:41.860 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:14:41.861 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:14:41.861 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:14:41.861 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:41.861 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:14:41.861 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:14:41.861 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:14:41.861 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:14:41.861 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:14:41.861 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.880+0000 7f308a11c640 1 librados: starting msgr at 2024-03-29T11:14:41.861 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.880+0000 7f308a11c640 1 librados: starting objecter 2024-03-29T11:14:41.861 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.880+0000 7f308a11c640 1 Processor -- start 2024-03-29T11:14:41.861 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.881+0000 7f308a11c640 1 -- start start 2024-03-29T11:14:41.861 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.881+0000 7f308a11c640 1 librados: setting wanted keys 2024-03-29T11:14:41.862 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.881+0000 7f308a11c640 1 librados: calling monclient init 2024-03-29T11:14:41.862 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.881+0000 7f308a11c640 10 monclient: init 2024-03-29T11:14:41.862 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.882+0000 7f308a11c640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:41.862 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.882+0000 7f308a11c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:14:41.862 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.882+0000 7f308a11c640 10 monclient(hunting): picked mon.noname-c con 0x7f30840ae250 addr v1:172.21.15.47:6790/0 2024-03-29T11:14:41.863 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.882+0000 7f308a11c640 10 monclient(hunting): picked mon.noname-a con 0x7f308413f970 addr v1:172.21.15.47:6789/0 2024-03-29T11:14:41.864 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.886+0000 7f308a11c640 10 monclient(hunting): picked mon.noname-b con 0x7f3084142800 addr v1:172.21.15.121:6789/0 2024-03-29T11:14:41.864 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.886+0000 7f308a11c640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3084145030 con 0x7f308413f970 2024-03-29T11:14:41.865 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.886+0000 7f308a11c640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3084138d70 con 0x7f3084142800 2024-03-29T11:14:41.865 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.886+0000 7f308a11c640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3084146100 con 0x7f30840ae250 2024-03-29T11:14:41.865 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.886+0000 7f308a11c640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:41.865 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.886+0000 7f308a11c640 10 monclient(hunting): authenticate will time out at 925.074828s 2024-03-29T11:14:41.867 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.888+0000 7f30837fe640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f30840ae250 0x7f30840ab820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:40907/0 (socket says 172.21.15.47:40907) 2024-03-29T11:14:41.867 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.888+0000 7f30837fe640 1 -- 172.21.15.47:0/924333592 learned_addr learned my addr 172.21.15.47:0/924333592 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:14:41.868 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.888+0000 7f306bfff640 10 client.?.objecter ms_handle_connect 0x7f30840ae250 2024-03-29T11:14:41.868 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.889+0000 7f306bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:41.868 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.889+0000 7f306bfff640 10 client.?.objecter ms_handle_connect 0x7f3084142800 2024-03-29T11:14:41.868 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.889+0000 7f306bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:41.868 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.889+0000 7f306bfff640 1 -- 172.21.15.47:0/924333592 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 387366880 0 0) 0x7f3084138d70 con 0x7f3084142800 2024-03-29T11:14:41.868 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.889+0000 7f306bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:14:41.868 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.889+0000 7f306bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:41.868 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.889+0000 7f306bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:41.868 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.889+0000 7f306bfff640 10 monclient(hunting): my global_id is 4300 2024-03-29T11:14:41.868 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.889+0000 7f306bfff640 1 -- 172.21.15.47:0/924333592 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3070004eb0 con 0x7f3084142800 2024-03-29T11:14:41.868 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.889+0000 7f306bfff640 1 -- 172.21.15.47:0/924333592 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 66975626 0 0) 0x7f3084146100 con 0x7f30840ae250 2024-03-29T11:14:41.868 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.889+0000 7f306bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:14:41.868 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.889+0000 7f306bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:41.868 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.889+0000 7f306bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:41.868 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.890+0000 7f306bfff640 10 monclient(hunting): my global_id is 4304 2024-03-29T11:14:41.869 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.890+0000 7f306bfff640 1 -- 172.21.15.47:0/924333592 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3084138d70 con 0x7f30840ae250 2024-03-29T11:14:41.869 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.890+0000 7f306bfff640 1 -- 172.21.15.47:0/924333592 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1976314852 0 0) 0x7f3070004eb0 con 0x7f3084142800 2024-03-29T11:14:41.869 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.890+0000 7f306bfff640 1 -- 172.21.15.47:0/924333592 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3084146100 con 0x7f3084142800 2024-03-29T11:14:41.869 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.890+0000 7f306bfff640 1 -- 172.21.15.47:0/924333592 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f3078001750 con 0x7f3084142800 2024-03-29T11:14:41.869 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.890+0000 7f306bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:41.869 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.890+0000 7f306bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:14:41.869 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.890+0000 7f306bfff640 10 monclient(hunting): dump: 2024-03-29T11:14:41.869 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:14:41.869 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:14:41.869 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:41.869 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:41.869 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:41.869 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:14:41.869 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:14:41.870 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:14:41.870 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:14:41.870 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:14:41.870 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.890+0000 7f306bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:14:41.874 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.890+0000 7f306bfff640 10 client.?.objecter ms_handle_connect 0x7f308413f970 2024-03-29T11:14:41.874 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.890+0000 7f306bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:41.874 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.890+0000 7f306bfff640 1 -- 172.21.15.47:0/924333592 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1425277323 0 0) 0x7f3084146100 con 0x7f3084142800 2024-03-29T11:14:41.874 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.890+0000 7f306bfff640 1 -- 172.21.15.47:0/924333592 >> v1:172.21.15.47:6790/0 conn(0x7f30840ae250 legacy=0x7f30840ab820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:41.874 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.891+0000 7f306bfff640 1 -- 172.21.15.47:0/924333592 >> v1:172.21.15.47:6789/0 conn(0x7f308413f970 legacy=0x7f3084137f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:41.874 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.891+0000 7f306bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:14:41.874 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.891+0000 7f306bfff640 1 monclient: found mon.b 2024-03-29T11:14:41.874 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.891+0000 7f306bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:41.875 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.891+0000 7f306bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:14:41.875 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.891+0000 7f306bfff640 1 -- 172.21.15.47:0/924333592 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30841463e0 con 0x7f3084142800 2024-03-29T11:14:41.875 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.891+0000 7f306bfff640 10 monclient: _finish_auth 0 2024-03-29T11:14:41.875 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.891+0000 7f306bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:14:41.875 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.891+0000 7f306bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:41.875 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.891+0000 7f308a11c640 5 monclient: authenticate success, global_id 4300 2024-03-29T11:14:41.875 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.891+0000 7f308a11c640 10 monclient: _renew_subs 2024-03-29T11:14:41.875 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.891+0000 7f308a11c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:14:41.875 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.891+0000 7f308a11c640 1 -- 172.21.15.47:0/924333592 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30841454c0 con 0x7f3084142800 2024-03-29T11:14:41.875 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.891+0000 7f308a11c640 10 client.4300.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:14:41.875 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.891+0000 7f308a11c640 10 monclient: _renew_subs 2024-03-29T11:14:41.875 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.891+0000 7f308a11c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:14:41.875 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.891+0000 7f308a11c640 1 -- 172.21.15.47:0/924333592 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f30840ac110 con 0x7f3084142800 2024-03-29T11:14:41.875 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.891+0000 7f308a11c640 1 librados: init done 2024-03-29T11:14:41.876 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.892+0000 7f306bfff640 1 -- 172.21.15.47:0/924333592 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3078001c60 con 0x7f3084142800 2024-03-29T11:14:41.876 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.892+0000 7f306bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:41.876 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.892+0000 7f306bfff640 1 -- 172.21.15.47:0/924333592 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f3078003790 con 0x7f3084142800 2024-03-29T11:14:41.876 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.892+0000 7f306bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:41.876 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.892+0000 7f306bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:14:41.876 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.892+0000 7f306bfff640 10 monclient: dump: 2024-03-29T11:14:41.876 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:14:41.876 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:14:41.876 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:41.876 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:41.876 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:41.876 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:14:41.876 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:14:41.876 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:14:41.876 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:14:41.877 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:14:41.877 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.892+0000 7f3081ffb640 4 set_mon_vals no callback set 2024-03-29T11:14:41.877 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.893+0000 7f308a11c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:41.877 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.893+0000 7f308a11c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:41.877 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.893+0000 7f308a11c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:14:41.877 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.893+0000 7f308a11c640 1 -- 172.21.15.47:0/924333592 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3064005080 con 0x7f3084142800 2024-03-29T11:14:41.877 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.894+0000 7f306bfff640 1 -- 172.21.15.47:0/924333592 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7f3078019760 con 0x7f3084142800 2024-03-29T11:14:41.877 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.895+0000 7f30837fe640 10 monclient: get_auth_request con 0x7f307006d9c0 auth_method 0 2024-03-29T11:14:41.877 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.895+0000 7f306bfff640 1 -- 172.21.15.47:0/924333592 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7f3078083650 con 0x7f3084142800 2024-03-29T11:14:41.877 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.895+0000 7f306bfff640 10 client.4300.objecter ms_dispatch 0x7f3084000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:14:41.877 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.895+0000 7f306bfff640 3 client.4300.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:14:41.877 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.895+0000 7f306bfff640 3 client.4300.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:14:41.877 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.895+0000 7f306bfff640 20 client.4300.objecter dump_active .. 0 homeless 2024-03-29T11:14:41.877 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.896+0000 7f306bfff640 10 client.4300.objecter ms_handle_connect 0x7f307006d9c0 2024-03-29T11:14:41.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.902+0000 7f306bfff640 1 -- 172.21.15.47:0/924333592 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f307804ef30 con 0x7f3084142800 2024-03-29T11:14:41.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.902+0000 7f306bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:41.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:41.902+0000 7f306bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:14:42.216 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:42.238+0000 7f308a11c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:42.216 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:42.238+0000 7f308a11c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:42.216 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:42.239+0000 7f308a11c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:14:42.217 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:42.239+0000 7f308a11c640 1 -- 172.21.15.47:0/924333592 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3064005330 con 0x7f3084142800 2024-03-29T11:14:42.217 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:42.239+0000 7f306bfff640 1 -- 172.21.15.47:0/924333592 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7f3078050e60 con 0x7f3084142800 2024-03-29T11:14:42.217 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:42.239+0000 7f306bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:42.217 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:42.239+0000 7f306bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:14:42.217 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:14:42.218 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:14:42.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:42.243+0000 7f308a11c640 10 librados: watch_flush enter 2024-03-29T11:14:42.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:42.246+0000 7f308a11c640 10 librados: watch_flush exit 2024-03-29T11:14:42.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:42.246+0000 7f308a11c640 20 client.4300.objecter shutdown clearing up homeless session... 2024-03-29T11:14:42.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:42.246+0000 7f308a11c640 10 client.4300.objecter successfully canceled tick 2024-03-29T11:14:42.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:42.246+0000 7f308a11c640 1 -- 172.21.15.47:0/924333592 >> v1:172.21.15.121:6816/1964117370 conn(0x7f307006d9c0 legacy=0x7f307006fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:42.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:42.246+0000 7f308a11c640 10 monclient: shutdown 2024-03-29T11:14:42.225 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:42.246+0000 7f308a11c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:42.225 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:42.246+0000 7f308a11c640 1 -- 172.21.15.47:0/924333592 >> v1:172.21.15.121:6789/0 conn(0x7f3084142800 legacy=0x7f3084138660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:42.225 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:42.246+0000 7f308a11c640 1 -- 172.21.15.47:0/924333592 shutdown_connections 2024-03-29T11:14:42.225 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:42.246+0000 7f308a11c640 1 -- 172.21.15.47:0/924333592 >> 172.21.15.47:0/924333592 conn(0x7f30840b35e0 msgr2=0x7f30840b3da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:42.225 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:42.247+0000 7f308a11c640 1 -- 172.21.15.47:0/924333592 shutdown_connections 2024-03-29T11:14:42.225 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:42.247+0000 7f308a11c640 1 -- 172.21.15.47:0/924333592 wait complete. 2024-03-29T11:14:42.225 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:42.247+0000 7f308a11c640 1 librados: shutdown 2024-03-29T11:14:42.244 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:14:49.056 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:14:49.220 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.240+0000 7f15f9d25640 10 monclient: get_monmap_and_config 2024-03-29T11:14:49.220 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.240+0000 7f15f9d25640 10 monclient: build_initial_monmap 2024-03-29T11:14:49.220 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.240+0000 7f15f9d25640 10 monclient: monmap: 2024-03-29T11:14:49.220 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:14:49.220 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:14:49.220 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:14:49.241601+0000 2024-03-29T11:14:49.221 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:14:49.241601+0000 2024-03-29T11:14:49.221 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:49.221 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:14:49.221 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:14:49.221 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:14:49.221 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:14:49.221 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:14:49.221 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.241+0000 7f15f9d25640 1 Processor -- start 2024-03-29T11:14:49.222 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.245+0000 7f15f9d25640 1 -- start start 2024-03-29T11:14:49.222 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.245+0000 7f15f9d25640 10 monclient: init 2024-03-29T11:14:49.223 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.246+0000 7f15f9d25640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:49.223 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.246+0000 7f15f9d25640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:14:49.223 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.246+0000 7f15f9d25640 10 monclient(hunting): picked mon.noname-c con 0x7f15f40aa430 addr v1:172.21.15.121:6789/0 2024-03-29T11:14:49.223 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.246+0000 7f15f9d25640 10 monclient(hunting): picked mon.noname-a con 0x7f15f40adfe0 addr v1:172.21.15.47:6789/0 2024-03-29T11:14:49.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.246+0000 7f15f9d25640 10 monclient(hunting): picked mon.noname-b con 0x7f15f4147fd0 addr v1:172.21.15.47:6790/0 2024-03-29T11:14:49.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.246+0000 7f15f9d25640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15f40ab140 con 0x7f15f40adfe0 2024-03-29T11:14:49.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.246+0000 7f15f9d25640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15f40ab700 con 0x7f15f40aa430 2024-03-29T11:14:49.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.246+0000 7f15f9d25640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15f413eeb0 con 0x7f15f4147fd0 2024-03-29T11:14:49.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.246+0000 7f15f9d25640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:49.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.246+0000 7f15f9d25640 10 monclient(hunting): authenticate will time out at 932.434249s 2024-03-29T11:14:49.229 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.250+0000 7f15f3fff640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f15f4147fd0 0x7f15f414a3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:59409/0 (socket says 172.21.15.47:59409) 2024-03-29T11:14:49.229 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.250+0000 7f15f3fff640 1 -- 172.21.15.47:0/2228400065 learned_addr learned my addr 172.21.15.47:0/2228400065 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:14:49.229 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.251+0000 7f15f27fc640 1 -- 172.21.15.47:0/2228400065 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3862679605 0 0) 0x7f15f413eeb0 con 0x7f15f4147fd0 2024-03-29T11:14:49.229 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.251+0000 7f15f27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:14:49.230 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.251+0000 7f15f27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:49.230 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.251+0000 7f15f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:49.230 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.252+0000 7f15f27fc640 10 monclient(hunting): my global_id is 4307 2024-03-29T11:14:49.230 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.252+0000 7f15f27fc640 1 -- 172.21.15.47:0/2228400065 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15dc003520 con 0x7f15f4147fd0 2024-03-29T11:14:49.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.252+0000 7f15f27fc640 1 -- 172.21.15.47:0/2228400065 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2409122194 0 0) 0x7f15dc003520 con 0x7f15f4147fd0 2024-03-29T11:14:49.233 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.252+0000 7f15f27fc640 1 -- 172.21.15.47:0/2228400065 >> v1:172.21.15.121:6789/0 conn(0x7f15f40aa430 legacy=0x7f15f413b9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:14:49.233 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.252+0000 7f15f27fc640 1 -- 172.21.15.47:0/2228400065 >> v1:172.21.15.47:6789/0 conn(0x7f15f40adfe0 legacy=0x7f15f40ae3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:14:49.233 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.252+0000 7f15f27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:14:49.233 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.252+0000 7f15f27fc640 1 monclient: found mon.noname-b 2024-03-29T11:14:49.233 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.253+0000 7f15f27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:49.233 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.253+0000 7f15f27fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-29T11:14:49.233 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.253+0000 7f15f27fc640 1 -- 172.21.15.47:0/2228400065 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15f40ab420 con 0x7f15f4147fd0 2024-03-29T11:14:49.233 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.253+0000 7f15f27fc640 10 monclient: _finish_auth 0 2024-03-29T11:14:49.233 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.253+0000 7f15f27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:14:49.233 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.253+0000 7f15f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:49.233 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.253+0000 7f15f27fc640 1 -- 172.21.15.47:0/2228400065 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f15ec001a10 con 0x7f15f4147fd0 2024-03-29T11:14:49.234 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.253+0000 7f15f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:49.234 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.253+0000 7f15f27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:14:49.234 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.253+0000 7f15f27fc640 10 monclient: dump: 2024-03-29T11:14:49.234 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:14:49.234 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:14:49.234 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:49.234 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:49.234 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:49.234 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:14:49.234 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:14:49.234 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:14:49.234 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:14:49.234 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:14:49.235 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.253+0000 7f15f9d25640 5 monclient: authenticate success, global_id 4307 2024-03-29T11:14:49.235 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.253+0000 7f15f9d25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:49.235 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.254+0000 7f15f27fc640 1 -- 172.21.15.47:0/2228400065 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f15ec001be0 con 0x7f15f4147fd0 2024-03-29T11:14:49.235 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.254+0000 7f15f27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:49.235 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.254+0000 7f15f27fc640 1 -- 172.21.15.47:0/2228400065 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f15ec002030 con 0x7f15f4147fd0 2024-03-29T11:14:49.235 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.254+0000 7f15f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:49.235 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.254+0000 7f15f27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:14:49.235 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.254+0000 7f15f27fc640 10 monclient: dump: 2024-03-29T11:14:49.235 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:14:49.235 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:14:49.235 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:49.235 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:49.235 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:49.235 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:14:49.235 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:14:49.236 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:14:49.236 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:14:49.236 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:14:49.236 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.255+0000 7f15f9d25640 10 monclient: get_monmap_and_config success 2024-03-29T11:14:49.236 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.255+0000 7f15f9d25640 4 set_mon_vals no callback set 2024-03-29T11:14:49.238 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.255+0000 7f15f9d25640 10 monclient: shutdown 2024-03-29T11:14:49.238 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.255+0000 7f15f9d25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:49.238 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.255+0000 7f15f9d25640 1 -- 172.21.15.47:0/2228400065 >> v1:172.21.15.47:6790/0 conn(0x7f15f4147fd0 legacy=0x7f15f414a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:49.238 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.256+0000 7f15f9d25640 1 -- 172.21.15.47:0/2228400065 shutdown_connections 2024-03-29T11:14:49.238 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.256+0000 7f15f9d25640 1 -- 172.21.15.47:0/2228400065 >> 172.21.15.47:0/2228400065 conn(0x7f15f40b35e0 msgr2=0x7f15f40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:49.239 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.256+0000 7f15f9d25640 1 -- 172.21.15.47:0/2228400065 shutdown_connections 2024-03-29T11:14:49.239 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.256+0000 7f15f9d25640 1 -- 172.21.15.47:0/2228400065 wait complete. 2024-03-29T11:14:49.239 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.257+0000 7f15f9d25640 10 monclient: build_initial_monmap 2024-03-29T11:14:49.239 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.257+0000 7f15f9d25640 10 monclient: monmap: 2024-03-29T11:14:49.239 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:14:49.239 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:14:49.239 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:14:49.239 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:14:49.239 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:49.239 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:14:49.239 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:14:49.240 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:14:49.240 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:14:49.240 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:14:49.240 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.257+0000 7f15f9d25640 1 librados: starting msgr at 2024-03-29T11:14:49.240 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.257+0000 7f15f9d25640 1 librados: starting objecter 2024-03-29T11:14:49.240 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.257+0000 7f15f9d25640 1 Processor -- start 2024-03-29T11:14:49.240 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.260+0000 7f15f9d25640 1 -- start start 2024-03-29T11:14:49.240 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.260+0000 7f15f9d25640 1 librados: setting wanted keys 2024-03-29T11:14:49.240 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.260+0000 7f15f9d25640 1 librados: calling monclient init 2024-03-29T11:14:49.240 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.260+0000 7f15f9d25640 10 monclient: init 2024-03-29T11:14:49.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.260+0000 7f15f9d25640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:49.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.260+0000 7f15f9d25640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:14:49.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.261+0000 7f15f9d25640 10 monclient(hunting): picked mon.noname-a con 0x7f15f40aa430 addr v1:172.21.15.47:6789/0 2024-03-29T11:14:49.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.261+0000 7f15f9d25640 10 monclient(hunting): picked mon.noname-b con 0x7f15f40adfe0 addr v1:172.21.15.121:6789/0 2024-03-29T11:14:49.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.261+0000 7f15f9d25640 10 monclient(hunting): picked mon.noname-c con 0x7f15f4147fd0 addr v1:172.21.15.47:6790/0 2024-03-29T11:14:49.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.261+0000 7f15f9d25640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15f4142450 con 0x7f15f40aa430 2024-03-29T11:14:49.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.261+0000 7f15f9d25640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15f413b0d0 con 0x7f15f40adfe0 2024-03-29T11:14:49.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.261+0000 7f15f9d25640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15f41419b0 con 0x7f15f4147fd0 2024-03-29T11:14:49.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.261+0000 7f15f9d25640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:49.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.261+0000 7f15f9d25640 10 monclient(hunting): authenticate will time out at 932.448999s 2024-03-29T11:14:49.242 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.264+0000 7f15f2ffd640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f15f40adfe0 0x7f15f413a2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.47:47047/0 (socket says 172.21.15.47:47047) 2024-03-29T11:14:49.242 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.264+0000 7f15f2ffd640 1 -- 172.21.15.47:0/3350345198 learned_addr learned my addr 172.21.15.47:0/3350345198 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:14:49.243 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.266+0000 7f15dbfff640 10 client.?.objecter ms_handle_connect 0x7f15f40adfe0 2024-03-29T11:14:49.243 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.266+0000 7f15dbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:49.244 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.266+0000 7f15dbfff640 1 -- 172.21.15.47:0/3350345198 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 546606694 0 0) 0x7f15f413b0d0 con 0x7f15f40adfe0 2024-03-29T11:14:49.246 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.267+0000 7f15dbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:14:49.246 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.267+0000 7f15dbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:49.246 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.267+0000 7f15dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:49.246 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.267+0000 7f15dbfff640 10 monclient(hunting): my global_id is 4303 2024-03-29T11:14:49.247 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.267+0000 7f15dbfff640 1 -- 172.21.15.47:0/3350345198 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15e0004eb0 con 0x7f15f40adfe0 2024-03-29T11:14:49.247 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.267+0000 7f15dbfff640 10 client.?.objecter ms_handle_connect 0x7f15f40aa430 2024-03-29T11:14:49.247 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.267+0000 7f15dbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:49.247 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.267+0000 7f15dbfff640 10 client.?.objecter ms_handle_connect 0x7f15f4147fd0 2024-03-29T11:14:49.247 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.267+0000 7f15dbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:49.247 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.267+0000 7f15dbfff640 1 -- 172.21.15.47:0/3350345198 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 510087220 0 0) 0x7f15f4142450 con 0x7f15f40aa430 2024-03-29T11:14:49.247 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.267+0000 7f15dbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:14:49.247 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.267+0000 7f15dbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:49.247 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.267+0000 7f15dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:49.247 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.267+0000 7f15dbfff640 10 monclient(hunting): my global_id is 4299 2024-03-29T11:14:49.247 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.267+0000 7f15dbfff640 1 -- 172.21.15.47:0/3350345198 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15f413b0d0 con 0x7f15f40aa430 2024-03-29T11:14:49.247 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.267+0000 7f15dbfff640 1 -- 172.21.15.47:0/3350345198 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1994524067 0 0) 0x7f15f41419b0 con 0x7f15f4147fd0 2024-03-29T11:14:49.247 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.267+0000 7f15dbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:14:49.247 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.267+0000 7f15dbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:49.248 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.267+0000 7f15dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:49.248 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.268+0000 7f15dbfff640 10 monclient(hunting): my global_id is 4310 2024-03-29T11:14:49.248 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.268+0000 7f15dbfff640 1 -- 172.21.15.47:0/3350345198 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15f4142450 con 0x7f15f4147fd0 2024-03-29T11:14:49.248 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.268+0000 7f15dbfff640 1 -- 172.21.15.47:0/3350345198 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1672068955 0 0) 0x7f15e0004eb0 con 0x7f15f40adfe0 2024-03-29T11:14:49.248 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.268+0000 7f15dbfff640 1 -- 172.21.15.47:0/3350345198 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15f41419b0 con 0x7f15f40adfe0 2024-03-29T11:14:49.248 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.268+0000 7f15dbfff640 1 -- 172.21.15.47:0/3350345198 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3583781476 0 0) 0x7f15f413b0d0 con 0x7f15f40aa430 2024-03-29T11:14:49.248 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.268+0000 7f15dbfff640 1 -- 172.21.15.47:0/3350345198 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15e0004eb0 con 0x7f15f40aa430 2024-03-29T11:14:49.248 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.268+0000 7f15dbfff640 1 -- 172.21.15.47:0/3350345198 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f15e4001960 con 0x7f15f40adfe0 2024-03-29T11:14:49.248 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.268+0000 7f15dbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:49.248 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.268+0000 7f15dbfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:14:49.248 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.268+0000 7f15dbfff640 10 monclient(hunting): dump: 2024-03-29T11:14:49.248 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:14:49.248 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:14:49.248 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:49.249 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:49.249 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:49.249 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:14:49.249 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:14:49.249 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:14:49.249 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:14:49.249 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:14:49.249 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.268+0000 7f15dbfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:14:49.249 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.268+0000 7f15dbfff640 1 -- 172.21.15.47:0/3350345198 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f15e8001790 con 0x7f15f40aa430 2024-03-29T11:14:49.249 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.268+0000 7f15dbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:49.249 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.268+0000 7f15dbfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:14:49.249 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.268+0000 7f15dbfff640 10 monclient(hunting): dump: 2024-03-29T11:14:49.249 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:14:49.249 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:14:49.250 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:49.250 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:49.250 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:49.250 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:14:49.250 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:14:49.250 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:14:49.250 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:14:49.250 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:14:49.252 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.270+0000 7f15dbfff640 1 -- 172.21.15.47:0/3350345198 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3096776949 0 0) 0x7f15f41419b0 con 0x7f15f40adfe0 2024-03-29T11:14:49.252 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.270+0000 7f15dbfff640 1 -- 172.21.15.47:0/3350345198 >> v1:172.21.15.47:6790/0 conn(0x7f15f4147fd0 legacy=0x7f15f413a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:49.252 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.270+0000 7f15dbfff640 1 -- 172.21.15.47:0/3350345198 >> v1:172.21.15.47:6789/0 conn(0x7f15f40aa430 legacy=0x7f15f41e4150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:49.253 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.270+0000 7f15dbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:14:49.253 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.270+0000 7f15dbfff640 1 monclient: found mon.b 2024-03-29T11:14:49.253 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.270+0000 7f15dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:49.253 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.270+0000 7f15dbfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:14:49.253 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.270+0000 7f15dbfff640 1 -- 172.21.15.47:0/3350345198 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15f4141c90 con 0x7f15f40adfe0 2024-03-29T11:14:49.253 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.270+0000 7f15dbfff640 10 monclient: _finish_auth 0 2024-03-29T11:14:49.253 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.270+0000 7f15dbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:14:49.253 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.270+0000 7f15dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:49.253 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.272+0000 7f15dbfff640 1 -- 172.21.15.47:0/3350345198 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f15e40035f0 con 0x7f15f40adfe0 2024-03-29T11:14:49.253 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.272+0000 7f15dbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:49.253 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.272+0000 7f15dbfff640 1 -- 172.21.15.47:0/3350345198 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f15e4003940 con 0x7f15f40adfe0 2024-03-29T11:14:49.253 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.272+0000 7f15f17fa640 4 set_mon_vals no callback set 2024-03-29T11:14:49.253 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.272+0000 7f15f9d25640 5 monclient: authenticate success, global_id 4303 2024-03-29T11:14:49.253 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.272+0000 7f15f9d25640 10 monclient: _renew_subs 2024-03-29T11:14:49.254 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.272+0000 7f15f9d25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:14:49.254 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.272+0000 7f15f9d25640 1 -- 172.21.15.47:0/3350345198 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15f41420b0 con 0x7f15f40adfe0 2024-03-29T11:14:49.254 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.272+0000 7f15f9d25640 10 client.4303.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:14:49.254 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.272+0000 7f15f9d25640 10 monclient: _renew_subs 2024-03-29T11:14:49.254 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.272+0000 7f15f9d25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:14:49.254 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.272+0000 7f15f9d25640 1 -- 172.21.15.47:0/3350345198 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f15f41e4a40 con 0x7f15f40adfe0 2024-03-29T11:14:49.254 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.272+0000 7f15f9d25640 1 librados: init done 2024-03-29T11:14:49.254 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.273+0000 7f15dbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:49.254 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.273+0000 7f15dbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:14:49.254 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.273+0000 7f15dbfff640 10 monclient: dump: 2024-03-29T11:14:49.254 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:14:49.254 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:14:49.254 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:49.254 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:49.255 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:49.255 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:14:49.255 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:14:49.255 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:14:49.255 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:14:49.255 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:14:49.255 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.273+0000 7f15f9d25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:49.255 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.273+0000 7f15f9d25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:49.255 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.273+0000 7f15f9d25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:14:49.255 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.273+0000 7f15f9d25640 1 -- 172.21.15.47:0/3350345198 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15f4075f10 con 0x7f15f40adfe0 2024-03-29T11:14:49.255 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.275+0000 7f15dbfff640 1 -- 172.21.15.47:0/3350345198 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7f15e4003aa0 con 0x7f15f40adfe0 2024-03-29T11:14:49.256 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.276+0000 7f15dbfff640 1 -- 172.21.15.47:0/3350345198 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7f15e4084840 con 0x7f15f40adfe0 2024-03-29T11:14:49.256 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.276+0000 7f15dbfff640 10 client.4303.objecter ms_dispatch 0x7f15f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:14:49.256 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.276+0000 7f15dbfff640 3 client.4303.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:14:49.256 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.276+0000 7f15dbfff640 3 client.4303.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:14:49.257 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.276+0000 7f15dbfff640 20 client.4303.objecter dump_active .. 0 homeless 2024-03-29T11:14:49.257 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.277+0000 7f15f37fe640 10 monclient: get_auth_request con 0x7f15e006e000 auth_method 0 2024-03-29T11:14:49.257 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.277+0000 7f15dbfff640 10 client.4303.objecter ms_handle_connect 0x7f15e006e000 2024-03-29T11:14:49.261 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.284+0000 7f15dbfff640 1 -- 172.21.15.47:0/3350345198 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f15e4050120 con 0x7f15f40adfe0 2024-03-29T11:14:49.261 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.284+0000 7f15dbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:49.262 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.284+0000 7f15dbfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:14:49.592 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.612+0000 7f15f9d25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:49.592 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.612+0000 7f15f9d25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:49.592 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.612+0000 7f15f9d25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:14:49.592 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.612+0000 7f15f9d25640 1 -- 172.21.15.47:0/3350345198 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f15f41e4a40 con 0x7f15f40adfe0 2024-03-29T11:14:49.592 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.614+0000 7f15dbfff640 1 -- 172.21.15.47:0/3350345198 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7f15f41e4a40 con 0x7f15f40adfe0 2024-03-29T11:14:49.592 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.614+0000 7f15dbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:49.593 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.614+0000 7f15dbfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:14:49.593 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:14:49.593 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:14:49.596 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.618+0000 7f15f9d25640 10 librados: watch_flush enter 2024-03-29T11:14:49.596 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.618+0000 7f15f9d25640 10 librados: watch_flush exit 2024-03-29T11:14:49.596 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.618+0000 7f15f9d25640 20 client.4303.objecter shutdown clearing up homeless session... 2024-03-29T11:14:49.596 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.618+0000 7f15f9d25640 10 client.4303.objecter successfully canceled tick 2024-03-29T11:14:49.597 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.619+0000 7f15f9d25640 1 -- 172.21.15.47:0/3350345198 >> v1:172.21.15.121:6816/1964117370 conn(0x7f15e006e000 legacy=0x7f15e00704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:49.597 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.619+0000 7f15f9d25640 10 monclient: shutdown 2024-03-29T11:14:49.597 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.619+0000 7f15f9d25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:49.597 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.619+0000 7f15f9d25640 1 -- 172.21.15.47:0/3350345198 >> v1:172.21.15.121:6789/0 conn(0x7f15f40adfe0 legacy=0x7f15f413a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:49.597 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.619+0000 7f15f9d25640 1 -- 172.21.15.47:0/3350345198 shutdown_connections 2024-03-29T11:14:49.597 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.619+0000 7f15f9d25640 1 -- 172.21.15.47:0/3350345198 >> 172.21.15.47:0/3350345198 conn(0x7f15f40b35e0 msgr2=0x7f15f40b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:49.600 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.622+0000 7f15f9d25640 1 -- 172.21.15.47:0/3350345198 shutdown_connections 2024-03-29T11:14:49.600 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.622+0000 7f15f9d25640 1 -- 172.21.15.47:0/3350345198 wait complete. 2024-03-29T11:14:49.600 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:49.623+0000 7f15f9d25640 1 librados: shutdown 2024-03-29T11:14:49.617 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:14:56.425 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:14:56.573 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.595+0000 7f65343ab640 10 monclient: get_monmap_and_config 2024-03-29T11:14:56.573 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.595+0000 7f65343ab640 10 monclient: build_initial_monmap 2024-03-29T11:14:56.573 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.595+0000 7f65343ab640 10 monclient: monmap: 2024-03-29T11:14:56.573 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:14:56.573 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:14:56.573 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:14:56.596779+0000 2024-03-29T11:14:56.573 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:14:56.596779+0000 2024-03-29T11:14:56.574 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:56.574 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:14:56.574 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:14:56.574 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:14:56.574 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:14:56.574 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:14:56.576 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.597+0000 7f65343ab640 1 Processor -- start 2024-03-29T11:14:56.576 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.597+0000 7f65343ab640 1 -- start start 2024-03-29T11:14:56.577 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.597+0000 7f65343ab640 10 monclient: init 2024-03-29T11:14:56.577 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.598+0000 7f65343ab640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:56.577 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.598+0000 7f65343ab640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:14:56.577 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.598+0000 7f65343ab640 10 monclient(hunting): picked mon.noname-a con 0x7f652c0b4330 addr v1:172.21.15.47:6789/0 2024-03-29T11:14:56.577 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.598+0000 7f65343ab640 10 monclient(hunting): picked mon.noname-c con 0x7f652c1470b0 addr v1:172.21.15.121:6789/0 2024-03-29T11:14:56.577 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.598+0000 7f65343ab640 10 monclient(hunting): picked mon.noname-b con 0x7f652c1439b0 addr v1:172.21.15.47:6790/0 2024-03-29T11:14:56.577 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.598+0000 7f65343ab640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f652c06f2a0 con 0x7f652c0b4330 2024-03-29T11:14:56.577 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.598+0000 7f65343ab640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f652c146b90 con 0x7f652c1470b0 2024-03-29T11:14:56.577 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.598+0000 7f65343ab640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f652c0b2ca0 con 0x7f652c1439b0 2024-03-29T11:14:56.577 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.598+0000 7f65343ab640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:56.577 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.598+0000 7f65343ab640 10 monclient(hunting): authenticate will time out at 939.786379s 2024-03-29T11:14:56.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.603+0000 7f6532147640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f652c0b4330 0x7f652c0b4710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:45235/0 (socket says 172.21.15.47:45235) 2024-03-29T11:14:56.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.603+0000 7f6532147640 1 -- 172.21.15.47:0/19432331 learned_addr learned my addr 172.21.15.47:0/19432331 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:14:56.584 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.604+0000 7f6531145640 1 -- 172.21.15.47:0/19432331 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3978344383 0 0) 0x7f652c06f2a0 con 0x7f652c0b4330 2024-03-29T11:14:56.584 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.604+0000 7f6531145640 10 monclient(hunting): _negotiate 2024-03-29T11:14:56.584 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.604+0000 7f6531145640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:56.584 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.604+0000 7f6531145640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:56.584 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.604+0000 7f6531145640 10 monclient(hunting): my global_id is 4302 2024-03-29T11:14:56.584 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.604+0000 7f6531145640 1 -- 172.21.15.47:0/19432331 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6518003520 con 0x7f652c0b4330 2024-03-29T11:14:56.588 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.606+0000 7f6531145640 1 -- 172.21.15.47:0/19432331 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2937487832 0 0) 0x7f652c146b90 con 0x7f652c1470b0 2024-03-29T11:14:56.588 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.606+0000 7f6531145640 10 monclient(hunting): _negotiate 2024-03-29T11:14:56.588 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.606+0000 7f6531145640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:56.588 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.606+0000 7f6531145640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:56.588 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.606+0000 7f6531145640 10 monclient(hunting): my global_id is 4306 2024-03-29T11:14:56.588 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.606+0000 7f6531145640 1 -- 172.21.15.47:0/19432331 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f652c06f2a0 con 0x7f652c1470b0 2024-03-29T11:14:56.588 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.607+0000 7f6531145640 1 -- 172.21.15.47:0/19432331 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1909605058 0 0) 0x7f652c06f2a0 con 0x7f652c1470b0 2024-03-29T11:14:56.588 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.607+0000 7f6531145640 1 -- 172.21.15.47:0/19432331 >> v1:172.21.15.47:6790/0 conn(0x7f652c1439b0 legacy=0x7f652c151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:56.588 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.607+0000 7f6531145640 1 -- 172.21.15.47:0/19432331 >> v1:172.21.15.47:6789/0 conn(0x7f652c0b4330 legacy=0x7f652c0b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:56.588 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.607+0000 7f6531145640 10 monclient: _finish_hunting 0 2024-03-29T11:14:56.588 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.607+0000 7f6531145640 1 monclient: found mon.noname-c 2024-03-29T11:14:56.588 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.607+0000 7f6531145640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:56.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.607+0000 7f6531145640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:14:56.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.607+0000 7f6531145640 1 -- 172.21.15.47:0/19432331 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f652c06e0c0 con 0x7f652c1470b0 2024-03-29T11:14:56.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.607+0000 7f6531145640 10 monclient: _finish_auth 0 2024-03-29T11:14:56.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.607+0000 7f6531145640 10 monclient: _check_auth_tickets 2024-03-29T11:14:56.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.607+0000 7f6531145640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:56.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.607+0000 7f6531145640 1 -- 172.21.15.47:0/19432331 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f651c0015f0 con 0x7f652c1470b0 2024-03-29T11:14:56.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.607+0000 7f6531145640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:56.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.607+0000 7f6531145640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:14:56.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.607+0000 7f6531145640 10 monclient: dump: 2024-03-29T11:14:56.589 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:14:56.589 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:14:56.589 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:56.589 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:56.589 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:56.590 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:14:56.590 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:14:56.590 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:14:56.590 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:14:56.590 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:14:56.590 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.607+0000 7f65343ab640 5 monclient: authenticate success, global_id 4306 2024-03-29T11:14:56.590 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.607+0000 7f65343ab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:56.590 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.607+0000 7f6531145640 1 -- 172.21.15.47:0/19432331 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f651c001750 con 0x7f652c1470b0 2024-03-29T11:14:56.590 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.607+0000 7f6531145640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:56.590 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.607+0000 7f6531145640 1 -- 172.21.15.47:0/19432331 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f651c001b60 con 0x7f652c1470b0 2024-03-29T11:14:56.590 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.607+0000 7f6531145640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:56.590 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.607+0000 7f6531145640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:14:56.590 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.607+0000 7f6531145640 10 monclient: dump: 2024-03-29T11:14:56.590 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:14:56.590 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:14:56.591 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:56.591 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:56.591 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:56.591 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:14:56.591 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:14:56.591 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:14:56.591 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:14:56.591 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:14:56.591 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.607+0000 7f65343ab640 10 monclient: get_monmap_and_config success 2024-03-29T11:14:56.591 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.607+0000 7f65343ab640 4 set_mon_vals no callback set 2024-03-29T11:14:56.591 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.608+0000 7f65343ab640 10 monclient: shutdown 2024-03-29T11:14:56.591 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.608+0000 7f65343ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:56.591 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.608+0000 7f65343ab640 1 -- 172.21.15.47:0/19432331 >> v1:172.21.15.121:6789/0 conn(0x7f652c1470b0 legacy=0x7f652c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:56.591 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.608+0000 7f65343ab640 1 -- 172.21.15.47:0/19432331 shutdown_connections 2024-03-29T11:14:56.592 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.608+0000 7f65343ab640 1 -- 172.21.15.47:0/19432331 >> 172.21.15.47:0/19432331 conn(0x7f652c066a00 msgr2=0x7f652c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:56.592 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.610+0000 7f65343ab640 1 -- 172.21.15.47:0/19432331 shutdown_connections 2024-03-29T11:14:56.592 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.610+0000 7f65343ab640 1 -- 172.21.15.47:0/19432331 wait complete. 2024-03-29T11:14:56.593 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.610+0000 7f65343ab640 10 monclient: build_initial_monmap 2024-03-29T11:14:56.593 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.610+0000 7f65343ab640 10 monclient: monmap: 2024-03-29T11:14:56.593 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:14:56.593 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:14:56.593 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:14:56.593 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:14:56.593 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:56.593 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:14:56.593 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:14:56.593 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:14:56.593 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:14:56.593 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:14:56.593 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.611+0000 7f65343ab640 1 librados: starting msgr at 2024-03-29T11:14:56.593 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.611+0000 7f65343ab640 1 librados: starting objecter 2024-03-29T11:14:56.594 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.611+0000 7f65343ab640 1 Processor -- start 2024-03-29T11:14:56.594 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.611+0000 7f65343ab640 1 -- start start 2024-03-29T11:14:56.594 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.611+0000 7f65343ab640 1 librados: setting wanted keys 2024-03-29T11:14:56.594 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.611+0000 7f65343ab640 1 librados: calling monclient init 2024-03-29T11:14:56.594 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.611+0000 7f65343ab640 10 monclient: init 2024-03-29T11:14:56.594 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.612+0000 7f65343ab640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:56.595 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.612+0000 7f65343ab640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:14:56.596 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.612+0000 7f65343ab640 10 monclient(hunting): picked mon.noname-c con 0x7f652c0b4330 addr v1:172.21.15.47:6790/0 2024-03-29T11:14:56.596 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.612+0000 7f65343ab640 10 monclient(hunting): picked mon.noname-b con 0x7f652c1439b0 addr v1:172.21.15.121:6789/0 2024-03-29T11:14:56.596 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.612+0000 7f65343ab640 10 monclient(hunting): picked mon.noname-a con 0x7f652c1470b0 addr v1:172.21.15.47:6789/0 2024-03-29T11:14:56.596 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.612+0000 7f65343ab640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f652c1df7b0 con 0x7f652c1470b0 2024-03-29T11:14:56.596 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.612+0000 7f65343ab640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f652c141890 con 0x7f652c1439b0 2024-03-29T11:14:56.596 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.612+0000 7f65343ab640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f652c1df9e0 con 0x7f652c0b4330 2024-03-29T11:14:56.596 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.612+0000 7f65343ab640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:56.596 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.612+0000 7f65343ab640 10 monclient(hunting): authenticate will time out at 939.800619s 2024-03-29T11:14:56.610 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.618+0000 7f6531946640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f652c1439b0 0x7f652c1400e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.47:39559/0 (socket says 172.21.15.47:39559) 2024-03-29T11:14:56.610 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.618+0000 7f6531946640 1 -- 172.21.15.47:0/461805382 learned_addr learned my addr 172.21.15.47:0/461805382 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:14:56.610 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.619+0000 7f6522ffd640 10 client.?.objecter ms_handle_connect 0x7f652c1439b0 2024-03-29T11:14:56.610 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.619+0000 7f6522ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:56.610 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.619+0000 7f6522ffd640 1 -- 172.21.15.47:0/461805382 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1692430786 0 0) 0x7f652c141890 con 0x7f652c1439b0 2024-03-29T11:14:56.610 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.619+0000 7f6522ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:14:56.610 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.619+0000 7f6522ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:56.610 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.619+0000 7f6522ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:56.610 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.619+0000 7f6522ffd640 10 monclient(hunting): my global_id is 4309 2024-03-29T11:14:56.610 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.619+0000 7f6522ffd640 1 -- 172.21.15.47:0/461805382 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6510004eb0 con 0x7f652c1439b0 2024-03-29T11:14:56.611 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.619+0000 7f6522ffd640 10 client.?.objecter ms_handle_connect 0x7f652c0b4330 2024-03-29T11:14:56.611 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.619+0000 7f6522ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:56.611 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.620+0000 7f6522ffd640 1 -- 172.21.15.47:0/461805382 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 571580108 0 0) 0x7f652c1df9e0 con 0x7f652c0b4330 2024-03-29T11:14:56.611 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.620+0000 7f6522ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:14:56.611 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.620+0000 7f6522ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:56.611 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.620+0000 7f6522ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:56.611 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.620+0000 7f6522ffd640 10 monclient(hunting): my global_id is 4316 2024-03-29T11:14:56.611 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.620+0000 7f6522ffd640 1 -- 172.21.15.47:0/461805382 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f652c141890 con 0x7f652c0b4330 2024-03-29T11:14:56.611 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.620+0000 7f6522ffd640 1 -- 172.21.15.47:0/461805382 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2476902864 0 0) 0x7f6510004eb0 con 0x7f652c1439b0 2024-03-29T11:14:56.611 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.620+0000 7f6522ffd640 1 -- 172.21.15.47:0/461805382 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f652c1df9e0 con 0x7f652c1439b0 2024-03-29T11:14:56.611 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.620+0000 7f6522ffd640 1 -- 172.21.15.47:0/461805382 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f651c001b60 con 0x7f652c1439b0 2024-03-29T11:14:56.611 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.620+0000 7f6522ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:56.611 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.620+0000 7f6522ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:14:56.611 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.620+0000 7f6522ffd640 10 monclient(hunting): dump: 2024-03-29T11:14:56.612 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:14:56.612 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:14:56.612 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:56.612 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:56.612 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:56.612 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:14:56.612 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:14:56.613 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:14:56.613 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:14:56.613 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:14:56.613 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.620+0000 7f6522ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:14:56.613 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.621+0000 7f6522ffd640 1 -- 172.21.15.47:0/461805382 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2412441808 0 0) 0x7f652c141890 con 0x7f652c0b4330 2024-03-29T11:14:56.613 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.621+0000 7f6522ffd640 1 -- 172.21.15.47:0/461805382 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6510004eb0 con 0x7f652c0b4330 2024-03-29T11:14:56.613 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.621+0000 7f6522ffd640 1 -- 172.21.15.47:0/461805382 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1118484431 0 0) 0x7f652c1df9e0 con 0x7f652c1439b0 2024-03-29T11:14:56.614 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.621+0000 7f6522ffd640 1 -- 172.21.15.47:0/461805382 >> v1:172.21.15.47:6790/0 conn(0x7f652c0b4330 legacy=0x7f652c1df0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:56.614 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.621+0000 7f6522ffd640 1 -- 172.21.15.47:0/461805382 >> v1:172.21.15.47:6789/0 conn(0x7f652c1470b0 legacy=0x7f652c125020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:14:56.614 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.621+0000 7f6522ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:14:56.614 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.621+0000 7f6522ffd640 1 monclient: found mon.b 2024-03-29T11:14:56.614 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.621+0000 7f6522ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:56.614 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.621+0000 7f6522ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:14:56.614 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.621+0000 7f6522ffd640 1 -- 172.21.15.47:0/461805382 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f652c140df0 con 0x7f652c1439b0 2024-03-29T11:14:56.614 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.621+0000 7f6522ffd640 10 monclient: _finish_auth 0 2024-03-29T11:14:56.614 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.621+0000 7f6522ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:14:56.614 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.621+0000 7f6522ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:56.614 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.621+0000 7f65343ab640 5 monclient: authenticate success, global_id 4309 2024-03-29T11:14:56.614 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.621+0000 7f65343ab640 10 monclient: _renew_subs 2024-03-29T11:14:56.614 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.621+0000 7f65343ab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:14:56.615 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.621+0000 7f65343ab640 1 -- 172.21.15.47:0/461805382 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f652c141040 con 0x7f652c1439b0 2024-03-29T11:14:56.615 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.621+0000 7f65343ab640 10 client.4309.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:14:56.615 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.621+0000 7f65343ab640 10 monclient: _renew_subs 2024-03-29T11:14:56.615 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.621+0000 7f65343ab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:14:56.615 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.621+0000 7f65343ab640 1 -- 172.21.15.47:0/461805382 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f652c1409d0 con 0x7f652c1439b0 2024-03-29T11:14:56.615 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.621+0000 7f65343ab640 1 librados: init done 2024-03-29T11:14:56.615 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.623+0000 7f6522ffd640 1 -- 172.21.15.47:0/461805382 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f651c0038a0 con 0x7f652c1439b0 2024-03-29T11:14:56.615 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.623+0000 7f6522ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:56.615 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.623+0000 7f6530944640 4 set_mon_vals no callback set 2024-03-29T11:14:56.615 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.626+0000 7f6522ffd640 1 -- 172.21.15.47:0/461805382 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f651c003c30 con 0x7f652c1439b0 2024-03-29T11:14:56.615 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.626+0000 7f6522ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:56.615 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.626+0000 7f6522ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:14:56.615 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.626+0000 7f6522ffd640 10 monclient: dump: 2024-03-29T11:14:56.615 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:14:56.615 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:14:56.616 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:56.616 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:14:56.616 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:56.616 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:14:56.616 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:14:56.616 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:14:56.616 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:14:56.616 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:14:56.616 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.626+0000 7f6522ffd640 1 -- 172.21.15.47:0/461805382 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7f651c003e50 con 0x7f652c1439b0 2024-03-29T11:14:56.616 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.626+0000 7f6522ffd640 1 -- 172.21.15.47:0/461805382 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7f651c084b00 con 0x7f652c1439b0 2024-03-29T11:14:56.616 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.626+0000 7f6522ffd640 10 client.4309.objecter ms_dispatch 0x7f652c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:14:56.616 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.626+0000 7f6522ffd640 3 client.4309.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:14:56.616 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.626+0000 7f6522ffd640 3 client.4309.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:14:56.616 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.626+0000 7f6522ffd640 20 client.4309.objecter dump_active .. 0 homeless 2024-03-29T11:14:56.617 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.627+0000 7f6532147640 10 monclient: get_auth_request con 0x7f651006dfa0 auth_method 0 2024-03-29T11:14:56.617 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.628+0000 7f65343ab640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:56.617 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.628+0000 7f65343ab640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:56.617 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.628+0000 7f65343ab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:14:56.617 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.628+0000 7f65343ab640 1 -- 172.21.15.47:0/461805382 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6514005080 con 0x7f652c1439b0 2024-03-29T11:14:56.617 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.630+0000 7f6522ffd640 10 client.4309.objecter ms_handle_connect 0x7f651006dfa0 2024-03-29T11:14:56.618 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.639+0000 7f6522ffd640 1 -- 172.21.15.47:0/461805382 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f651c050460 con 0x7f652c1439b0 2024-03-29T11:14:56.618 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.639+0000 7f6522ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:56.618 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.639+0000 7f6522ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:14:56.974 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.996+0000 7f65343ab640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:56.974 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.996+0000 7f65343ab640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:56.974 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.996+0000 7f65343ab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:14:56.974 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.996+0000 7f65343ab640 1 -- 172.21.15.47:0/461805382 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6514005330 con 0x7f652c1439b0 2024-03-29T11:14:56.975 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.997+0000 7f6522ffd640 1 -- 172.21.15.47:0/461805382 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7f651c052390 con 0x7f652c1439b0 2024-03-29T11:14:56.975 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.997+0000 7f6522ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:56.975 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:56.997+0000 7f6522ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:14:56.975 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:14:56.975 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:14:56.979 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:57.001+0000 7f65343ab640 10 librados: watch_flush enter 2024-03-29T11:14:56.979 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:57.001+0000 7f65343ab640 10 librados: watch_flush exit 2024-03-29T11:14:56.979 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:57.001+0000 7f65343ab640 20 client.4309.objecter shutdown clearing up homeless session... 2024-03-29T11:14:56.979 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:57.001+0000 7f65343ab640 10 client.4309.objecter successfully canceled tick 2024-03-29T11:14:56.979 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:57.001+0000 7f65343ab640 1 -- 172.21.15.47:0/461805382 >> v1:172.21.15.121:6816/1964117370 conn(0x7f651006dfa0 legacy=0x7f6510070460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:56.979 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:57.001+0000 7f65343ab640 10 monclient: shutdown 2024-03-29T11:14:56.979 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:57.001+0000 7f65343ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:56.979 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:57.001+0000 7f65343ab640 1 -- 172.21.15.47:0/461805382 >> v1:172.21.15.121:6789/0 conn(0x7f652c1439b0 legacy=0x7f652c1400e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:56.979 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:57.001+0000 7f65343ab640 1 -- 172.21.15.47:0/461805382 shutdown_connections 2024-03-29T11:14:56.979 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:57.001+0000 7f65343ab640 1 -- 172.21.15.47:0/461805382 >> 172.21.15.47:0/461805382 conn(0x7f652c066a00 msgr2=0x7f652c070dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:56.979 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:57.001+0000 7f65343ab640 1 -- 172.21.15.47:0/461805382 shutdown_connections 2024-03-29T11:14:56.979 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:57.001+0000 7f65343ab640 1 -- 172.21.15.47:0/461805382 wait complete. 2024-03-29T11:14:56.980 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:14:57.001+0000 7f65343ab640 1 librados: shutdown 2024-03-29T11:14:56.994 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:15:03.800 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:15:03.952 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.975+0000 7fe277f04640 10 monclient: get_monmap_and_config 2024-03-29T11:15:03.953 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.975+0000 7fe277f04640 10 monclient: build_initial_monmap 2024-03-29T11:15:03.953 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.975+0000 7fe277f04640 10 monclient: monmap: 2024-03-29T11:15:03.953 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:15:03.953 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:15:03.953 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:15:03.976170+0000 2024-03-29T11:15:03.953 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:15:03.976170+0000 2024-03-29T11:15:03.953 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:03.953 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:15:03.953 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:15:03.953 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:15:03.953 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:15:03.953 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:15:03.954 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.976+0000 7fe277f04640 1 Processor -- start 2024-03-29T11:15:03.954 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.976+0000 7fe277f04640 1 -- start start 2024-03-29T11:15:03.954 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.976+0000 7fe277f04640 10 monclient: init 2024-03-29T11:15:03.957 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.977+0000 7fe277f04640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:03.958 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.977+0000 7fe277f04640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:15:03.958 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.977+0000 7fe275ca0640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fe2701457f0 0x7fe27013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:38832/0 (socket says 172.21.15.47:38832) 2024-03-29T11:15:03.958 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.977+0000 7fe275ca0640 1 -- 172.21.15.47:0/2942028217 learned_addr learned my addr 172.21.15.47:0/2942028217 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:15:03.958 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.977+0000 7fe277f04640 10 monclient(hunting): picked mon.noname-a con 0x7fe2701457f0 addr v1:172.21.15.47:6789/0 2024-03-29T11:15:03.958 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.977+0000 7fe277f04640 10 monclient(hunting): picked mon.noname-c con 0x7fe27013f650 addr v1:172.21.15.121:6789/0 2024-03-29T11:15:03.958 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.978+0000 7fe277f04640 10 monclient(hunting): picked mon.noname-b con 0x7fe2700a9d70 addr v1:172.21.15.47:6790/0 2024-03-29T11:15:03.958 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.978+0000 7fe277f04640 1 -- 172.21.15.47:0/2942028217 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2700aaef0 con 0x7fe2701457f0 2024-03-29T11:15:03.958 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.978+0000 7fe277f04640 1 -- 172.21.15.47:0/2942028217 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe270145bd0 con 0x7fe27013f650 2024-03-29T11:15:03.958 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.978+0000 7fe277f04640 1 -- 172.21.15.47:0/2942028217 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe270145eb0 con 0x7fe2700a9d70 2024-03-29T11:15:03.958 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.978+0000 7fe277f04640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:03.958 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.978+0000 7fe277f04640 10 monclient(hunting): authenticate will time out at 947.166154s 2024-03-29T11:15:03.959 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.978+0000 7fe274c9e640 1 -- 172.21.15.47:0/2942028217 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2537424583 0 0) 0x7fe270145eb0 con 0x7fe2700a9d70 2024-03-29T11:15:03.959 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.978+0000 7fe274c9e640 10 monclient(hunting): _negotiate 2024-03-29T11:15:03.959 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.978+0000 7fe274c9e640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:03.959 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.978+0000 7fe274c9e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:03.959 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.978+0000 7fe274c9e640 10 monclient(hunting): my global_id is 4319 2024-03-29T11:15:03.959 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.978+0000 7fe274c9e640 1 -- 172.21.15.47:0/2942028217 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe25c003520 con 0x7fe2700a9d70 2024-03-29T11:15:03.959 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.978+0000 7fe274c9e640 1 -- 172.21.15.47:0/2942028217 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1108626242 0 0) 0x7fe2700aaef0 con 0x7fe2701457f0 2024-03-29T11:15:03.959 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.978+0000 7fe274c9e640 10 monclient(hunting): _negotiate 2024-03-29T11:15:03.959 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.978+0000 7fe274c9e640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:03.959 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.978+0000 7fe274c9e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:03.959 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.978+0000 7fe274c9e640 10 monclient(hunting): my global_id is 4305 2024-03-29T11:15:03.959 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.978+0000 7fe274c9e640 1 -- 172.21.15.47:0/2942028217 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe270145eb0 con 0x7fe2701457f0 2024-03-29T11:15:03.959 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.979+0000 7fe274c9e640 1 -- 172.21.15.47:0/2942028217 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1009788450 0 0) 0x7fe25c003520 con 0x7fe2700a9d70 2024-03-29T11:15:03.959 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.979+0000 7fe274c9e640 1 -- 172.21.15.47:0/2942028217 >> v1:172.21.15.121:6789/0 conn(0x7fe27013f650 legacy=0x7fe2700b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:03.960 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.979+0000 7fe274c9e640 1 -- 172.21.15.47:0/2942028217 >> v1:172.21.15.47:6789/0 conn(0x7fe2701457f0 legacy=0x7fe27013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:03.960 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.979+0000 7fe274c9e640 10 monclient: _finish_hunting 0 2024-03-29T11:15:03.960 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.979+0000 7fe274c9e640 1 monclient: found mon.noname-b 2024-03-29T11:15:03.960 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.979+0000 7fe274c9e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:03.960 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.979+0000 7fe274c9e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-29T11:15:03.960 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.979+0000 7fe274c9e640 1 -- 172.21.15.47:0/2942028217 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe270146340 con 0x7fe2700a9d70 2024-03-29T11:15:03.960 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.979+0000 7fe274c9e640 10 monclient: _finish_auth 0 2024-03-29T11:15:03.960 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.979+0000 7fe274c9e640 10 monclient: _check_auth_tickets 2024-03-29T11:15:03.960 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.979+0000 7fe274c9e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:03.961 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.979+0000 7fe274c9e640 1 -- 172.21.15.47:0/2942028217 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fe268001a10 con 0x7fe2700a9d70 2024-03-29T11:15:03.961 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.979+0000 7fe274c9e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:03.961 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.979+0000 7fe274c9e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:15:03.961 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.979+0000 7fe274c9e640 10 monclient: dump: 2024-03-29T11:15:03.961 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:15:03.961 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:15:03.961 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:03.962 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:03.962 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:03.962 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:15:03.962 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:15:03.962 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:15:03.962 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:15:03.962 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:15:03.962 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.979+0000 7fe277f04640 5 monclient: authenticate success, global_id 4319 2024-03-29T11:15:03.962 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.979+0000 7fe277f04640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:15:03.962 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.979+0000 7fe274c9e640 1 -- 172.21.15.47:0/2942028217 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe268001c30 con 0x7fe2700a9d70 2024-03-29T11:15:03.962 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.979+0000 7fe274c9e640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:03.963 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.979+0000 7fe274c9e640 1 -- 172.21.15.47:0/2942028217 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fe268001fc0 con 0x7fe2700a9d70 2024-03-29T11:15:03.963 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.979+0000 7fe277f04640 10 monclient: get_monmap_and_config success 2024-03-29T11:15:03.963 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.979+0000 7fe277f04640 4 set_mon_vals no callback set 2024-03-29T11:15:03.963 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.980+0000 7fe274c9e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:03.963 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.980+0000 7fe274c9e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:15:03.963 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.980+0000 7fe274c9e640 10 monclient: dump: 2024-03-29T11:15:03.963 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:15:03.963 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:15:03.963 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:03.963 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:03.963 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:03.963 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:15:03.963 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:15:03.964 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:15:03.964 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:15:03.964 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:15:03.968 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.980+0000 7fe277f04640 10 monclient: shutdown 2024-03-29T11:15:03.969 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.980+0000 7fe277f04640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:03.969 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.980+0000 7fe277f04640 1 -- 172.21.15.47:0/2942028217 >> v1:172.21.15.47:6790/0 conn(0x7fe2700a9d70 legacy=0x7fe27014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:03.969 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.981+0000 7fe277f04640 1 -- 172.21.15.47:0/2942028217 shutdown_connections 2024-03-29T11:15:03.969 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.981+0000 7fe277f04640 1 -- 172.21.15.47:0/2942028217 >> 172.21.15.47:0/2942028217 conn(0x7fe270066a00 msgr2=0x7fe270066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:03.969 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.981+0000 7fe277f04640 1 -- 172.21.15.47:0/2942028217 shutdown_connections 2024-03-29T11:15:03.969 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.981+0000 7fe277f04640 1 -- 172.21.15.47:0/2942028217 wait complete. 2024-03-29T11:15:03.969 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.982+0000 7fe277f04640 10 monclient: build_initial_monmap 2024-03-29T11:15:03.969 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.982+0000 7fe277f04640 10 monclient: monmap: 2024-03-29T11:15:03.969 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:15:03.969 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:15:03.969 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:15:03.969 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:15:03.969 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:03.970 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:15:03.970 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:15:03.970 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:15:03.970 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:15:03.970 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:15:03.970 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.983+0000 7fe277f04640 1 librados: starting msgr at 2024-03-29T11:15:03.970 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.983+0000 7fe277f04640 1 librados: starting objecter 2024-03-29T11:15:03.970 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.983+0000 7fe277f04640 1 Processor -- start 2024-03-29T11:15:03.970 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.983+0000 7fe277f04640 1 -- start start 2024-03-29T11:15:03.970 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.983+0000 7fe277f04640 1 librados: setting wanted keys 2024-03-29T11:15:03.970 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.983+0000 7fe277f04640 1 librados: calling monclient init 2024-03-29T11:15:03.970 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.983+0000 7fe277f04640 10 monclient: init 2024-03-29T11:15:03.970 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.984+0000 7fe277f04640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:03.970 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.984+0000 7fe277f04640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:15:03.971 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.984+0000 7fe277f04640 10 monclient(hunting): picked mon.noname-a con 0x7fe2700a9d70 addr v1:172.21.15.47:6789/0 2024-03-29T11:15:03.971 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.984+0000 7fe277f04640 10 monclient(hunting): picked mon.noname-b con 0x7fe27013f650 addr v1:172.21.15.121:6789/0 2024-03-29T11:15:03.971 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.984+0000 7fe275ca0640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fe2700a9d70 0x7fe2701cb340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:39154/0 (socket says 172.21.15.47:39154) 2024-03-29T11:15:03.971 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.984+0000 7fe277f04640 10 monclient(hunting): picked mon.noname-c con 0x7fe2701457f0 addr v1:172.21.15.47:6790/0 2024-03-29T11:15:03.971 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.984+0000 7fe277f04640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2701db480 con 0x7fe2700a9d70 2024-03-29T11:15:03.971 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.984+0000 7fe277f04640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2701cd070 con 0x7fe27013f650 2024-03-29T11:15:03.971 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.984+0000 7fe277f04640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2701db910 con 0x7fe2701457f0 2024-03-29T11:15:03.971 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.984+0000 7fe277f04640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:03.971 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.984+0000 7fe277f04640 10 monclient(hunting): authenticate will time out at 947.172724s 2024-03-29T11:15:03.971 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.984+0000 7fe275ca0640 1 -- 172.21.15.47:0/3845590975 learned_addr learned my addr 172.21.15.47:0/3845590975 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:15:03.971 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.985+0000 7fe2667fc640 10 client.?.objecter ms_handle_connect 0x7fe2700a9d70 2024-03-29T11:15:03.971 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.985+0000 7fe2667fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:03.971 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.985+0000 7fe2667fc640 10 client.?.objecter ms_handle_connect 0x7fe2701457f0 2024-03-29T11:15:03.971 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.985+0000 7fe2667fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:03.972 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.985+0000 7fe2667fc640 10 client.?.objecter ms_handle_connect 0x7fe27013f650 2024-03-29T11:15:03.972 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.985+0000 7fe2667fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:03.972 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.985+0000 7fe2667fc640 1 -- 172.21.15.47:0/3845590975 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3697368221 0 0) 0x7fe2701db480 con 0x7fe2700a9d70 2024-03-29T11:15:03.972 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.985+0000 7fe2667fc640 10 monclient(hunting): _negotiate 2024-03-29T11:15:03.972 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.985+0000 7fe2667fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:03.972 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.985+0000 7fe2667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:03.972 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.985+0000 7fe2667fc640 10 monclient(hunting): my global_id is 4308 2024-03-29T11:15:03.972 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.985+0000 7fe2667fc640 1 -- 172.21.15.47:0/3845590975 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe254004eb0 con 0x7fe2700a9d70 2024-03-29T11:15:03.972 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.986+0000 7fe2667fc640 1 -- 172.21.15.47:0/3845590975 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4256493518 0 0) 0x7fe2701db910 con 0x7fe2701457f0 2024-03-29T11:15:03.972 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.986+0000 7fe2667fc640 10 monclient(hunting): _negotiate 2024-03-29T11:15:03.972 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.986+0000 7fe2667fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:03.972 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.986+0000 7fe2667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:03.972 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.986+0000 7fe2667fc640 10 monclient(hunting): my global_id is 4322 2024-03-29T11:15:03.972 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.986+0000 7fe2667fc640 1 -- 172.21.15.47:0/3845590975 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2701db480 con 0x7fe2701457f0 2024-03-29T11:15:03.973 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.986+0000 7fe2667fc640 1 -- 172.21.15.47:0/3845590975 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2310811759 0 0) 0x7fe254004eb0 con 0x7fe2700a9d70 2024-03-29T11:15:03.973 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.986+0000 7fe2667fc640 1 -- 172.21.15.47:0/3845590975 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2701db910 con 0x7fe2700a9d70 2024-03-29T11:15:03.973 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.986+0000 7fe2667fc640 1 -- 172.21.15.47:0/3845590975 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fe26c001890 con 0x7fe2700a9d70 2024-03-29T11:15:03.973 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.986+0000 7fe2667fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:03.973 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.986+0000 7fe2667fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:15:03.973 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.986+0000 7fe2667fc640 10 monclient(hunting): dump: 2024-03-29T11:15:03.973 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:15:03.973 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:15:03.973 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:03.973 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:03.973 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:03.973 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:15:03.973 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:15:03.973 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:15:03.974 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:15:03.974 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:15:03.974 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.986+0000 7fe2667fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:15:03.974 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.986+0000 7fe2667fc640 1 -- 172.21.15.47:0/3845590975 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1158074377 0 0) 0x7fe2701cd070 con 0x7fe27013f650 2024-03-29T11:15:03.974 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.986+0000 7fe2667fc640 10 monclient(hunting): _negotiate 2024-03-29T11:15:03.974 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.986+0000 7fe2667fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:03.974 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.986+0000 7fe2667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:03.974 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.986+0000 7fe2667fc640 10 monclient(hunting): my global_id is 4312 2024-03-29T11:15:03.974 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.986+0000 7fe2667fc640 1 -- 172.21.15.47:0/3845590975 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe254004eb0 con 0x7fe27013f650 2024-03-29T11:15:03.974 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.986+0000 7fe2667fc640 1 -- 172.21.15.47:0/3845590975 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4017107485 0 0) 0x7fe2701db910 con 0x7fe2700a9d70 2024-03-29T11:15:03.974 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.986+0000 7fe2667fc640 1 -- 172.21.15.47:0/3845590975 >> v1:172.21.15.47:6790/0 conn(0x7fe2701457f0 legacy=0x7fe2701e2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:03.974 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.986+0000 7fe2667fc640 1 -- 172.21.15.47:0/3845590975 >> v1:172.21.15.121:6789/0 conn(0x7fe27013f650 legacy=0x7fe270137dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:03.974 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.986+0000 7fe2667fc640 10 monclient: _finish_hunting 0 2024-03-29T11:15:03.974 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.986+0000 7fe2667fc640 1 monclient: found mon.a 2024-03-29T11:15:03.975 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.986+0000 7fe2667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:03.975 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.986+0000 7fe2667fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:15:03.975 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.986+0000 7fe2667fc640 1 -- 172.21.15.47:0/3845590975 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2700b1660 con 0x7fe2700a9d70 2024-03-29T11:15:03.975 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.986+0000 7fe2667fc640 10 monclient: _finish_auth 0 2024-03-29T11:15:03.975 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.986+0000 7fe2667fc640 10 monclient: _check_auth_tickets 2024-03-29T11:15:03.975 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.986+0000 7fe2667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:03.975 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.987+0000 7fe277f04640 5 monclient: authenticate success, global_id 4308 2024-03-29T11:15:03.975 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.987+0000 7fe277f04640 10 monclient: _renew_subs 2024-03-29T11:15:03.975 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.987+0000 7fe277f04640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:15:03.975 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.987+0000 7fe2667fc640 1 -- 172.21.15.47:0/3845590975 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe26c0047e0 con 0x7fe2700a9d70 2024-03-29T11:15:03.975 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.987+0000 7fe277f04640 1 -- 172.21.15.47:0/3845590975 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2701cba50 con 0x7fe2700a9d70 2024-03-29T11:15:03.975 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.987+0000 7fe277f04640 10 client.4308.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:15:03.975 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.987+0000 7fe2667fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:03.975 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.987+0000 7fe277f04640 10 monclient: _renew_subs 2024-03-29T11:15:03.976 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.987+0000 7fe277f04640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:15:03.976 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.987+0000 7fe277f04640 1 -- 172.21.15.47:0/3845590975 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2701cbd50 con 0x7fe2700a9d70 2024-03-29T11:15:03.976 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.987+0000 7fe277f04640 1 librados: init done 2024-03-29T11:15:03.976 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.987+0000 7fe267fff640 4 set_mon_vals no callback set 2024-03-29T11:15:03.976 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.987+0000 7fe2667fc640 1 -- 172.21.15.47:0/3845590975 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fe26c004af0 con 0x7fe2700a9d70 2024-03-29T11:15:03.976 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.988+0000 7fe2667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:03.976 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.988+0000 7fe2667fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:15:03.976 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.988+0000 7fe2667fc640 10 monclient: dump: 2024-03-29T11:15:03.976 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:15:03.976 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:15:03.976 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:03.976 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:03.976 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:03.976 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:15:03.977 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:15:03.977 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:15:03.977 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:15:03.977 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:15:03.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.989+0000 7fe277f04640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:03.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.989+0000 7fe277f04640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:03.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.989+0000 7fe277f04640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:15:03.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.989+0000 7fe277f04640 1 -- 172.21.15.47:0/3845590975 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2700b1850 con 0x7fe2700a9d70 2024-03-29T11:15:03.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.989+0000 7fe2667fc640 1 -- 172.21.15.47:0/3845590975 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7fe26c01aac0 con 0x7fe2700a9d70 2024-03-29T11:15:03.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.989+0000 7fe2667fc640 1 -- 172.21.15.47:0/3845590975 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7fe26c0849c0 con 0x7fe2700a9d70 2024-03-29T11:15:03.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.989+0000 7fe2667fc640 10 client.4308.objecter ms_dispatch 0x7fe270000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:15:03.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.989+0000 7fe2667fc640 3 client.4308.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:15:03.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.989+0000 7fe2667fc640 3 client.4308.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:15:03.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.990+0000 7fe27549f640 10 monclient: get_auth_request con 0x7fe25406de80 auth_method 0 2024-03-29T11:15:03.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.990+0000 7fe2667fc640 20 client.4308.objecter dump_active .. 0 homeless 2024-03-29T11:15:03.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.990+0000 7fe2667fc640 10 client.4308.objecter ms_handle_connect 0x7fe25406de80 2024-03-29T11:15:03.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.998+0000 7fe2667fc640 1 -- 172.21.15.47:0/3845590975 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe26c050300 con 0x7fe2700a9d70 2024-03-29T11:15:03.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.998+0000 7fe2667fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:03.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:03.998+0000 7fe2667fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:15:04.349 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:04.371+0000 7fe277f04640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:04.350 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:04.371+0000 7fe277f04640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:04.350 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:04.371+0000 7fe277f04640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:15:04.350 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:04.371+0000 7fe277f04640 1 -- 172.21.15.47:0/3845590975 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe270075f10 con 0x7fe2700a9d70 2024-03-29T11:15:04.351 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:04.373+0000 7fe2667fc640 1 -- 172.21.15.47:0/3845590975 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7fe26c084070 con 0x7fe2700a9d70 2024-03-29T11:15:04.351 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:04.373+0000 7fe2667fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:04.351 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:04.373+0000 7fe2667fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:15:04.351 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:15:04.352 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:15:04.356 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:04.376+0000 7fe277f04640 10 librados: watch_flush enter 2024-03-29T11:15:04.356 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:04.376+0000 7fe277f04640 10 librados: watch_flush exit 2024-03-29T11:15:04.356 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:04.376+0000 7fe277f04640 20 client.4308.objecter shutdown clearing up homeless session... 2024-03-29T11:15:04.356 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:04.376+0000 7fe277f04640 10 client.4308.objecter successfully canceled tick 2024-03-29T11:15:04.356 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:04.376+0000 7fe277f04640 1 -- 172.21.15.47:0/3845590975 >> v1:172.21.15.121:6816/1964117370 conn(0x7fe25406de80 legacy=0x7fe254070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:04.357 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:04.377+0000 7fe277f04640 10 monclient: shutdown 2024-03-29T11:15:04.357 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:04.377+0000 7fe277f04640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:04.357 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:04.377+0000 7fe277f04640 1 -- 172.21.15.47:0/3845590975 >> v1:172.21.15.47:6789/0 conn(0x7fe2700a9d70 legacy=0x7fe2701cb340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:04.357 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:04.377+0000 7fe277f04640 1 -- 172.21.15.47:0/3845590975 shutdown_connections 2024-03-29T11:15:04.357 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:04.377+0000 7fe277f04640 1 -- 172.21.15.47:0/3845590975 >> 172.21.15.47:0/3845590975 conn(0x7fe270066a00 msgr2=0x7fe2700b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:04.357 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:04.377+0000 7fe277f04640 1 -- 172.21.15.47:0/3845590975 shutdown_connections 2024-03-29T11:15:04.357 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:04.377+0000 7fe277f04640 1 -- 172.21.15.47:0/3845590975 wait complete. 2024-03-29T11:15:04.357 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:04.377+0000 7fe277f04640 1 librados: shutdown 2024-03-29T11:15:04.369 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:15:11.176 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:15:11.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.333+0000 7f0e79faf640 10 monclient: get_monmap_and_config 2024-03-29T11:15:11.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.333+0000 7f0e79faf640 10 monclient: build_initial_monmap 2024-03-29T11:15:11.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.333+0000 7f0e79faf640 10 monclient: monmap: 2024-03-29T11:15:11.311 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:15:11.311 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:15:11.311 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:15:11.334416+0000 2024-03-29T11:15:11.311 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:15:11.334416+0000 2024-03-29T11:15:11.311 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:11.311 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:15:11.312 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:15:11.312 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:15:11.312 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:15:11.312 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:15:11.314 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.334+0000 7f0e79faf640 1 Processor -- start 2024-03-29T11:15:11.314 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.334+0000 7f0e79faf640 1 -- start start 2024-03-29T11:15:11.314 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.334+0000 7f0e79faf640 10 monclient: init 2024-03-29T11:15:11.314 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.335+0000 7f0e79faf640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:11.314 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.335+0000 7f0e79faf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:15:11.314 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.335+0000 7f0e79faf640 10 monclient(hunting): picked mon.noname-c con 0x7f0e741457f0 addr v1:172.21.15.121:6789/0 2024-03-29T11:15:11.314 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.335+0000 7f0e79faf640 10 monclient(hunting): picked mon.noname-b con 0x7f0e7413f650 addr v1:172.21.15.47:6790/0 2024-03-29T11:15:11.315 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.335+0000 7f0e79faf640 10 monclient(hunting): picked mon.noname-a con 0x7f0e740a9d70 addr v1:172.21.15.47:6789/0 2024-03-29T11:15:11.315 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.335+0000 7f0e79faf640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e740aaef0 con 0x7f0e740a9d70 2024-03-29T11:15:11.315 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.335+0000 7f0e79faf640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e74145bd0 con 0x7f0e741457f0 2024-03-29T11:15:11.315 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.335+0000 7f0e737fe640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f0e741457f0 0x7f0e7413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.47:44408/0 (socket says 172.21.15.47:44408) 2024-03-29T11:15:11.315 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.335+0000 7f0e737fe640 1 -- 172.21.15.47:0/1663146440 learned_addr learned my addr 172.21.15.47:0/1663146440 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:15:11.315 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.335+0000 7f0e79faf640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e74145eb0 con 0x7f0e7413f650 2024-03-29T11:15:11.315 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.335+0000 7f0e79faf640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:11.315 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.336+0000 7f0e79faf640 10 monclient(hunting): authenticate will time out at 954.523936s 2024-03-29T11:15:11.315 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.336+0000 7f0e727fc640 1 -- 172.21.15.47:0/1663146440 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3434394287 0 0) 0x7f0e74145bd0 con 0x7f0e741457f0 2024-03-29T11:15:11.315 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.336+0000 7f0e727fc640 10 monclient(hunting): _negotiate 2024-03-29T11:15:11.315 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.336+0000 7f0e727fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:11.315 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.336+0000 7f0e727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:11.315 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.336+0000 7f0e727fc640 10 monclient(hunting): my global_id is 4315 2024-03-29T11:15:11.315 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.336+0000 7f0e727fc640 1 -- 172.21.15.47:0/1663146440 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e5c003520 con 0x7f0e741457f0 2024-03-29T11:15:11.316 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.336+0000 7f0e727fc640 1 -- 172.21.15.47:0/1663146440 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2414813209 0 0) 0x7f0e5c003520 con 0x7f0e741457f0 2024-03-29T11:15:11.316 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.337+0000 7f0e727fc640 1 -- 172.21.15.47:0/1663146440 >> v1:172.21.15.47:6790/0 conn(0x7f0e7413f650 legacy=0x7f0e740b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:11.316 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.337+0000 7f0e727fc640 1 -- 172.21.15.47:0/1663146440 >> v1:172.21.15.47:6789/0 conn(0x7f0e740a9d70 legacy=0x7f0e7414d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:15:11.317 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.338+0000 7f0e727fc640 10 monclient: _finish_hunting 0 2024-03-29T11:15:11.317 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.338+0000 7f0e727fc640 1 monclient: found mon.noname-c 2024-03-29T11:15:11.317 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.338+0000 7f0e727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:11.318 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.338+0000 7f0e727fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:15:11.318 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.339+0000 7f0e727fc640 1 -- 172.21.15.47:0/1663146440 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e74146340 con 0x7f0e741457f0 2024-03-29T11:15:11.318 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.339+0000 7f0e727fc640 10 monclient: _finish_auth 0 2024-03-29T11:15:11.318 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.339+0000 7f0e727fc640 10 monclient: _check_auth_tickets 2024-03-29T11:15:11.318 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.339+0000 7f0e727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:11.318 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.339+0000 7f0e727fc640 1 -- 172.21.15.47:0/1663146440 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f0e64001690 con 0x7f0e741457f0 2024-03-29T11:15:11.318 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.339+0000 7f0e79faf640 5 monclient: authenticate success, global_id 4315 2024-03-29T11:15:11.318 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.339+0000 7f0e79faf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:15:11.318 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.339+0000 7f0e727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:11.318 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.339+0000 7f0e727fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:15:11.318 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.339+0000 7f0e727fc640 10 monclient: dump: 2024-03-29T11:15:11.318 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:15:11.318 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:15:11.318 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:11.319 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:11.319 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:11.319 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:15:11.319 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:15:11.319 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:15:11.319 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:15:11.319 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:15:11.319 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.339+0000 7f0e79faf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:15:11.320 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.341+0000 7f0e727fc640 1 -- 172.21.15.47:0/1663146440 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0e64001920 con 0x7f0e741457f0 2024-03-29T11:15:11.320 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.341+0000 7f0e727fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:11.320 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.341+0000 7f0e727fc640 1 -- 172.21.15.47:0/1663146440 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f0e64001cb0 con 0x7f0e741457f0 2024-03-29T11:15:11.321 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.341+0000 7f0e727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:11.321 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.341+0000 7f0e727fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:15:11.321 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.341+0000 7f0e727fc640 10 monclient: dump: 2024-03-29T11:15:11.321 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:15:11.321 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:15:11.321 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:11.321 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:11.321 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:11.321 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:15:11.321 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:15:11.321 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:15:11.321 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:15:11.321 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:15:11.321 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.341+0000 7f0e79faf640 10 monclient: get_monmap_and_config success 2024-03-29T11:15:11.321 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.341+0000 7f0e79faf640 4 set_mon_vals no callback set 2024-03-29T11:15:11.322 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.342+0000 7f0e79faf640 10 monclient: shutdown 2024-03-29T11:15:11.322 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.342+0000 7f0e79faf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:11.322 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.342+0000 7f0e79faf640 1 -- 172.21.15.47:0/1663146440 >> v1:172.21.15.121:6789/0 conn(0x7f0e741457f0 legacy=0x7f0e7413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:11.322 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.342+0000 7f0e79faf640 1 -- 172.21.15.47:0/1663146440 shutdown_connections 2024-03-29T11:15:11.322 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.342+0000 7f0e79faf640 1 -- 172.21.15.47:0/1663146440 >> 172.21.15.47:0/1663146440 conn(0x7f0e74066a00 msgr2=0x7f0e74066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:11.322 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.342+0000 7f0e79faf640 1 -- 172.21.15.47:0/1663146440 shutdown_connections 2024-03-29T11:15:11.322 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.342+0000 7f0e79faf640 1 -- 172.21.15.47:0/1663146440 wait complete. 2024-03-29T11:15:11.322 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.343+0000 7f0e79faf640 10 monclient: build_initial_monmap 2024-03-29T11:15:11.322 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.343+0000 7f0e79faf640 10 monclient: monmap: 2024-03-29T11:15:11.322 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:15:11.322 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:15:11.322 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:15:11.322 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:15:11.322 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:11.323 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:15:11.323 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:15:11.323 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:15:11.323 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:15:11.323 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:15:11.324 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.343+0000 7f0e79faf640 1 librados: starting msgr at 2024-03-29T11:15:11.325 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.343+0000 7f0e79faf640 1 librados: starting objecter 2024-03-29T11:15:11.325 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.343+0000 7f0e79faf640 1 Processor -- start 2024-03-29T11:15:11.325 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.344+0000 7f0e79faf640 1 -- start start 2024-03-29T11:15:11.325 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.344+0000 7f0e79faf640 1 librados: setting wanted keys 2024-03-29T11:15:11.325 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.344+0000 7f0e79faf640 1 librados: calling monclient init 2024-03-29T11:15:11.325 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.344+0000 7f0e79faf640 10 monclient: init 2024-03-29T11:15:11.325 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.344+0000 7f0e79faf640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:11.325 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.345+0000 7f0e79faf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:15:11.325 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.345+0000 7f0e79faf640 10 monclient(hunting): picked mon.noname-c con 0x7f0e740a9d70 addr v1:172.21.15.47:6790/0 2024-03-29T11:15:11.325 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.345+0000 7f0e79faf640 10 monclient(hunting): picked mon.noname-b con 0x7f0e7413f650 addr v1:172.21.15.121:6789/0 2024-03-29T11:15:11.325 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.345+0000 7f0e737fe640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f0e740a9d70 0x7f0e74136790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:55302/0 (socket says 172.21.15.47:55302) 2024-03-29T11:15:11.325 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.345+0000 7f0e737fe640 1 -- 172.21.15.47:0/4076175597 learned_addr learned my addr 172.21.15.47:0/4076175597 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:15:11.325 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.346+0000 7f0e5bfff640 10 client.?.objecter ms_handle_connect 0x7f0e740a9d70 2024-03-29T11:15:11.326 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.346+0000 7f0e79faf640 10 monclient(hunting): picked mon.noname-a con 0x7f0e741457f0 addr v1:172.21.15.47:6789/0 2024-03-29T11:15:11.326 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.346+0000 7f0e79faf640 1 -- 172.21.15.47:0/4076175597 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e74135d90 con 0x7f0e741457f0 2024-03-29T11:15:11.326 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.346+0000 7f0e79faf640 1 -- 172.21.15.47:0/4076175597 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e74140db0 con 0x7f0e7413f650 2024-03-29T11:15:11.326 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.346+0000 7f0e79faf640 1 -- 172.21.15.47:0/4076175597 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e74136000 con 0x7f0e740a9d70 2024-03-29T11:15:11.326 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.346+0000 7f0e79faf640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:11.326 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.346+0000 7f0e79faf640 10 monclient(hunting): authenticate will time out at 954.534142s 2024-03-29T11:15:11.326 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.346+0000 7f0e5bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:11.326 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.346+0000 7f0e5bfff640 10 client.?.objecter ms_handle_connect 0x7f0e7413f650 2024-03-29T11:15:11.326 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.346+0000 7f0e5bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:11.326 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.346+0000 7f0e5bfff640 1 -- 172.21.15.47:0/4076175597 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2498896396 0 0) 0x7f0e74140db0 con 0x7f0e7413f650 2024-03-29T11:15:11.326 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.346+0000 7f0e5bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:15:11.326 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.346+0000 7f0e5bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:11.326 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.346+0000 7f0e5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:11.327 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.346+0000 7f0e5bfff640 10 monclient(hunting): my global_id is 4318 2024-03-29T11:15:11.327 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.346+0000 7f0e5bfff640 1 -- 172.21.15.47:0/4076175597 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e60004eb0 con 0x7f0e7413f650 2024-03-29T11:15:11.327 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.347+0000 7f0e5bfff640 1 -- 172.21.15.47:0/4076175597 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 299198841 0 0) 0x7f0e74136000 con 0x7f0e740a9d70 2024-03-29T11:15:11.327 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.347+0000 7f0e5bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:15:11.327 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.347+0000 7f0e5bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:11.327 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.347+0000 7f0e5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:11.327 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.347+0000 7f0e5bfff640 10 monclient(hunting): my global_id is 4325 2024-03-29T11:15:11.327 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.347+0000 7f0e5bfff640 1 -- 172.21.15.47:0/4076175597 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e74140db0 con 0x7f0e740a9d70 2024-03-29T11:15:11.331 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.347+0000 7f0e5bfff640 1 -- 172.21.15.47:0/4076175597 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1239297576 0 0) 0x7f0e60004eb0 con 0x7f0e7413f650 2024-03-29T11:15:11.331 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.347+0000 7f0e5bfff640 1 -- 172.21.15.47:0/4076175597 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e74136000 con 0x7f0e7413f650 2024-03-29T11:15:11.331 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.347+0000 7f0e5bfff640 1 -- 172.21.15.47:0/4076175597 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f0e68001930 con 0x7f0e7413f650 2024-03-29T11:15:11.331 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.348+0000 7f0e5bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:11.332 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.348+0000 7f0e5bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:15:11.332 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.348+0000 7f0e5bfff640 10 monclient(hunting): dump: 2024-03-29T11:15:11.332 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:15:11.332 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:15:11.332 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:11.332 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:11.332 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:11.332 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:15:11.332 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:15:11.332 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:15:11.332 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:15:11.332 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:15:11.332 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.348+0000 7f0e5bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:15:11.332 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.348+0000 7f0e5bfff640 1 -- 172.21.15.47:0/4076175597 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1549872661 0 0) 0x7f0e74140db0 con 0x7f0e740a9d70 2024-03-29T11:15:11.333 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.348+0000 7f0e5bfff640 1 -- 172.21.15.47:0/4076175597 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e60004eb0 con 0x7f0e740a9d70 2024-03-29T11:15:11.333 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.348+0000 7f0e5bfff640 1 -- 172.21.15.47:0/4076175597 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f0e64001610 con 0x7f0e740a9d70 2024-03-29T11:15:11.333 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.348+0000 7f0e5bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:11.333 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.348+0000 7f0e5bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:15:11.333 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.348+0000 7f0e5bfff640 10 monclient(hunting): dump: 2024-03-29T11:15:11.333 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:15:11.333 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:15:11.333 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:11.333 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:11.333 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:11.333 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:15:11.333 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:15:11.333 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:15:11.333 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:15:11.334 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:15:11.334 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.348+0000 7f0e5bfff640 1 -- 172.21.15.47:0/4076175597 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2706651348 0 0) 0x7f0e74136000 con 0x7f0e7413f650 2024-03-29T11:15:11.334 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.348+0000 7f0e5bfff640 1 -- 172.21.15.47:0/4076175597 >> v1:172.21.15.47:6790/0 conn(0x7f0e740a9d70 legacy=0x7f0e74136790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:11.334 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.348+0000 7f0e5bfff640 1 -- 172.21.15.47:0/4076175597 >> v1:172.21.15.47:6789/0 conn(0x7f0e741457f0 legacy=0x7f0e741e6bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:11.334 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.348+0000 7f0e5bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:15:11.334 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.348+0000 7f0e5bfff640 1 monclient: found mon.b 2024-03-29T11:15:11.334 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.348+0000 7f0e5bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:11.334 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.348+0000 7f0e5bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:15:11.334 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.348+0000 7f0e5bfff640 1 -- 172.21.15.47:0/4076175597 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e741346b0 con 0x7f0e7413f650 2024-03-29T11:15:11.334 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.348+0000 7f0e5bfff640 10 monclient: _finish_auth 0 2024-03-29T11:15:11.334 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.348+0000 7f0e5bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:15:11.334 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.348+0000 7f0e5bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:11.334 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.348+0000 7f0e79faf640 5 monclient: authenticate success, global_id 4318 2024-03-29T11:15:11.334 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.348+0000 7f0e79faf640 10 monclient: _renew_subs 2024-03-29T11:15:11.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.348+0000 7f0e79faf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:15:11.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.348+0000 7f0e79faf640 1 -- 172.21.15.47:0/4076175597 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e74142ca0 con 0x7f0e7413f650 2024-03-29T11:15:11.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.348+0000 7f0e79faf640 10 client.4318.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:15:11.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.349+0000 7f0e5bfff640 1 -- 172.21.15.47:0/4076175597 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0e680035f0 con 0x7f0e7413f650 2024-03-29T11:15:11.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.349+0000 7f0e79faf640 10 monclient: _renew_subs 2024-03-29T11:15:11.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.349+0000 7f0e79faf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:15:11.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.349+0000 7f0e79faf640 1 -- 172.21.15.47:0/4076175597 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e74134bc0 con 0x7f0e7413f650 2024-03-29T11:15:11.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.349+0000 7f0e79faf640 1 librados: init done 2024-03-29T11:15:11.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.349+0000 7f0e5bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:11.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.349+0000 7f0e5bfff640 1 -- 172.21.15.47:0/4076175597 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f0e68003980 con 0x7f0e7413f650 2024-03-29T11:15:11.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.349+0000 7f0e71ffb640 4 set_mon_vals no callback set 2024-03-29T11:15:11.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.349+0000 7f0e5bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:11.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.349+0000 7f0e5bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:15:11.336 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.349+0000 7f0e5bfff640 10 monclient: dump: 2024-03-29T11:15:11.336 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:15:11.336 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:15:11.336 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:11.336 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:11.336 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:11.336 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:15:11.336 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:15:11.336 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:15:11.336 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:15:11.336 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:15:11.336 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.350+0000 7f0e79faf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:11.336 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.350+0000 7f0e79faf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:11.336 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.350+0000 7f0e79faf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:15:11.336 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.350+0000 7f0e79faf640 1 -- 172.21.15.47:0/4076175597 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e54005080 con 0x7f0e7413f650 2024-03-29T11:15:11.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.351+0000 7f0e5bfff640 1 -- 172.21.15.47:0/4076175597 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7f0e68019950 con 0x7f0e7413f650 2024-03-29T11:15:11.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.352+0000 7f0e5bfff640 1 -- 172.21.15.47:0/4076175597 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7f0e68083860 con 0x7f0e7413f650 2024-03-29T11:15:11.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.352+0000 7f0e5bfff640 10 client.4318.objecter ms_dispatch 0x7f0e74000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:15:11.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.352+0000 7f0e5bfff640 3 client.4318.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:15:11.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.352+0000 7f0e5bfff640 3 client.4318.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:15:11.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.352+0000 7f0e5bfff640 20 client.4318.objecter dump_active .. 0 homeless 2024-03-29T11:15:11.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.353+0000 7f0e737fe640 10 monclient: get_auth_request con 0x7f0e6006de80 auth_method 0 2024-03-29T11:15:11.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.353+0000 7f0e5bfff640 10 client.4318.objecter ms_handle_connect 0x7f0e6006de80 2024-03-29T11:15:11.338 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.358+0000 7f0e5bfff640 1 -- 172.21.15.47:0/4076175597 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0e6804f140 con 0x7f0e7413f650 2024-03-29T11:15:11.338 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.358+0000 7f0e5bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:11.338 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.358+0000 7f0e5bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:15:11.644 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.666+0000 7f0e79faf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:11.644 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.666+0000 7f0e79faf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:11.644 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.666+0000 7f0e79faf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:15:11.644 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.666+0000 7f0e79faf640 1 -- 172.21.15.47:0/4076175597 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0e54005330 con 0x7f0e7413f650 2024-03-29T11:15:11.645 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.667+0000 7f0e5bfff640 1 -- 172.21.15.47:0/4076175597 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7f0e68051070 con 0x7f0e7413f650 2024-03-29T11:15:11.645 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.667+0000 7f0e5bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:11.645 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.667+0000 7f0e5bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:15:11.645 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:15:11.645 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:15:11.649 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.671+0000 7f0e79faf640 10 librados: watch_flush enter 2024-03-29T11:15:11.649 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.671+0000 7f0e79faf640 10 librados: watch_flush exit 2024-03-29T11:15:11.649 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.671+0000 7f0e79faf640 20 client.4318.objecter shutdown clearing up homeless session... 2024-03-29T11:15:11.650 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.671+0000 7f0e79faf640 10 client.4318.objecter successfully canceled tick 2024-03-29T11:15:11.650 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.671+0000 7f0e79faf640 1 -- 172.21.15.47:0/4076175597 >> v1:172.21.15.121:6816/1964117370 conn(0x7f0e6006de80 legacy=0x7f0e60070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:11.650 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.671+0000 7f0e79faf640 10 monclient: shutdown 2024-03-29T11:15:11.650 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.671+0000 7f0e79faf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:11.650 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.671+0000 7f0e79faf640 1 -- 172.21.15.47:0/4076175597 >> v1:172.21.15.121:6789/0 conn(0x7f0e7413f650 legacy=0x7f0e741e34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:11.650 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.672+0000 7f0e79faf640 1 -- 172.21.15.47:0/4076175597 shutdown_connections 2024-03-29T11:15:11.650 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.672+0000 7f0e79faf640 1 -- 172.21.15.47:0/4076175597 >> 172.21.15.47:0/4076175597 conn(0x7f0e74066a00 msgr2=0x7f0e7413ac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:11.651 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.672+0000 7f0e79faf640 1 -- 172.21.15.47:0/4076175597 shutdown_connections 2024-03-29T11:15:11.651 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.672+0000 7f0e79faf640 1 -- 172.21.15.47:0/4076175597 wait complete. 2024-03-29T11:15:11.651 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:11.673+0000 7f0e79faf640 1 librados: shutdown 2024-03-29T11:15:11.663 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:15:18.471 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:15:18.604 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.626+0000 7faae4451640 10 monclient: get_monmap_and_config 2024-03-29T11:15:18.604 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.626+0000 7faae4451640 10 monclient: build_initial_monmap 2024-03-29T11:15:18.604 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.626+0000 7faae4451640 10 monclient: monmap: 2024-03-29T11:15:18.604 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:15:18.604 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:15:18.604 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:15:18.627465+0000 2024-03-29T11:15:18.604 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:15:18.627465+0000 2024-03-29T11:15:18.604 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:18.604 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:15:18.604 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:15:18.604 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:15:18.604 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:15:18.605 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:15:18.605 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.627+0000 7faae4451640 1 Processor -- start 2024-03-29T11:15:18.606 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.627+0000 7faae4451640 1 -- start start 2024-03-29T11:15:18.606 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.627+0000 7faae4451640 10 monclient: init 2024-03-29T11:15:18.606 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.628+0000 7faae4451640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:18.606 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.628+0000 7faae4451640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:15:18.606 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.628+0000 7faae4451640 10 monclient(hunting): picked mon.noname-c con 0x7faadc1457f0 addr v1:172.21.15.121:6789/0 2024-03-29T11:15:18.608 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.629+0000 7faae21ed640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7faadc1457f0 0x7faadc13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.47:44134/0 (socket says 172.21.15.47:44134) 2024-03-29T11:15:18.608 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.629+0000 7faae4451640 10 monclient(hunting): picked mon.noname-a con 0x7faadc13f650 addr v1:172.21.15.47:6789/0 2024-03-29T11:15:18.608 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.629+0000 7faae4451640 10 monclient(hunting): picked mon.noname-b con 0x7faadc0a9d70 addr v1:172.21.15.47:6790/0 2024-03-29T11:15:18.608 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.629+0000 7faae4451640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faadc0aaef0 con 0x7faadc13f650 2024-03-29T11:15:18.608 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.629+0000 7faae4451640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faadc145bd0 con 0x7faadc1457f0 2024-03-29T11:15:18.608 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.629+0000 7faae4451640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faadc145eb0 con 0x7faadc0a9d70 2024-03-29T11:15:18.608 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.629+0000 7faae4451640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:18.608 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.629+0000 7faae4451640 10 monclient(hunting): authenticate will time out at 961.817558s 2024-03-29T11:15:18.608 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.629+0000 7faae21ed640 1 -- 172.21.15.47:0/2028320954 learned_addr learned my addr 172.21.15.47:0/2028320954 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:15:18.608 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.630+0000 7faae11eb640 1 -- 172.21.15.47:0/2028320954 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4077878802 0 0) 0x7faadc145bd0 con 0x7faadc1457f0 2024-03-29T11:15:18.608 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.630+0000 7faae11eb640 10 monclient(hunting): _negotiate 2024-03-29T11:15:18.608 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.630+0000 7faae11eb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:18.609 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.630+0000 7faae11eb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:18.609 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.630+0000 7faae11eb640 10 monclient(hunting): my global_id is 4321 2024-03-29T11:15:18.609 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.630+0000 7faae11eb640 1 -- 172.21.15.47:0/2028320954 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faac8003520 con 0x7faadc1457f0 2024-03-29T11:15:18.609 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.630+0000 7faae11eb640 1 -- 172.21.15.47:0/2028320954 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1852185449 0 0) 0x7faadc145eb0 con 0x7faadc0a9d70 2024-03-29T11:15:18.609 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.630+0000 7faae11eb640 10 monclient(hunting): _negotiate 2024-03-29T11:15:18.609 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.630+0000 7faae11eb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:18.609 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.630+0000 7faae11eb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:18.609 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.630+0000 7faae11eb640 10 monclient(hunting): my global_id is 4328 2024-03-29T11:15:18.609 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.630+0000 7faae11eb640 1 -- 172.21.15.47:0/2028320954 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faadc145bd0 con 0x7faadc0a9d70 2024-03-29T11:15:18.609 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.630+0000 7faae11eb640 1 -- 172.21.15.47:0/2028320954 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 484636948 0 0) 0x7faadc145bd0 con 0x7faadc0a9d70 2024-03-29T11:15:18.609 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.630+0000 7faae11eb640 1 -- 172.21.15.47:0/2028320954 >> v1:172.21.15.121:6789/0 conn(0x7faadc1457f0 legacy=0x7faadc13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:18.609 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.630+0000 7faae11eb640 1 -- 172.21.15.47:0/2028320954 >> v1:172.21.15.47:6789/0 conn(0x7faadc13f650 legacy=0x7faadc0b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:15:18.612 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.631+0000 7faae11eb640 10 monclient: _finish_hunting 0 2024-03-29T11:15:18.612 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.631+0000 7faae11eb640 1 monclient: found mon.noname-b 2024-03-29T11:15:18.612 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.631+0000 7faae11eb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:18.612 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.631+0000 7faae11eb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-29T11:15:18.612 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.631+0000 7faae11eb640 1 -- 172.21.15.47:0/2028320954 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faadc146340 con 0x7faadc0a9d70 2024-03-29T11:15:18.613 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.631+0000 7faae11eb640 10 monclient: _finish_auth 0 2024-03-29T11:15:18.613 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.631+0000 7faae11eb640 10 monclient: _check_auth_tickets 2024-03-29T11:15:18.613 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.631+0000 7faae11eb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:18.613 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.631+0000 7faae11eb640 1 -- 172.21.15.47:0/2028320954 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7faad40019f0 con 0x7faadc0a9d70 2024-03-29T11:15:18.613 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.631+0000 7faae11eb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:18.613 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.632+0000 7faae11eb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:15:18.613 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.632+0000 7faae11eb640 10 monclient: dump: 2024-03-29T11:15:18.613 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:15:18.613 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:15:18.613 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:18.613 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:18.613 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:18.613 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:15:18.613 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:15:18.613 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:15:18.614 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:15:18.614 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:15:18.614 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.632+0000 7faae4451640 5 monclient: authenticate success, global_id 4328 2024-03-29T11:15:18.614 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.632+0000 7faae4451640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:15:18.614 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.632+0000 7faae11eb640 1 -- 172.21.15.47:0/2028320954 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faad4001bd0 con 0x7faadc0a9d70 2024-03-29T11:15:18.614 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.632+0000 7faae11eb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:18.614 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.632+0000 7faae4451640 10 monclient: get_monmap_and_config success 2024-03-29T11:15:18.614 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.632+0000 7faae4451640 4 set_mon_vals no callback set 2024-03-29T11:15:18.614 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.632+0000 7faae11eb640 1 -- 172.21.15.47:0/2028320954 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7faad4002020 con 0x7faadc0a9d70 2024-03-29T11:15:18.614 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.632+0000 7faae11eb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:18.614 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.632+0000 7faae11eb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:15:18.614 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.632+0000 7faae11eb640 10 monclient: dump: 2024-03-29T11:15:18.614 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:15:18.614 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:15:18.615 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:18.615 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:18.615 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:18.615 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:15:18.615 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:15:18.615 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:15:18.615 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:15:18.615 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:15:18.615 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.633+0000 7faae4451640 10 monclient: shutdown 2024-03-29T11:15:18.615 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.633+0000 7faae4451640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:18.615 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.633+0000 7faae4451640 1 -- 172.21.15.47:0/2028320954 >> v1:172.21.15.47:6790/0 conn(0x7faadc0a9d70 legacy=0x7faadc14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:18.615 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.633+0000 7faae4451640 1 -- 172.21.15.47:0/2028320954 shutdown_connections 2024-03-29T11:15:18.615 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.633+0000 7faae4451640 1 -- 172.21.15.47:0/2028320954 >> 172.21.15.47:0/2028320954 conn(0x7faadc066a00 msgr2=0x7faadc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:18.615 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.633+0000 7faae4451640 1 -- 172.21.15.47:0/2028320954 shutdown_connections 2024-03-29T11:15:18.615 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.633+0000 7faae4451640 1 -- 172.21.15.47:0/2028320954 wait complete. 2024-03-29T11:15:18.615 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.634+0000 7faae4451640 10 monclient: build_initial_monmap 2024-03-29T11:15:18.616 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.634+0000 7faae4451640 10 monclient: monmap: 2024-03-29T11:15:18.616 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:15:18.616 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:15:18.616 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:15:18.616 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:15:18.616 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:18.616 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:15:18.616 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:15:18.616 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:15:18.616 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:15:18.616 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:15:18.616 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.635+0000 7faae4451640 1 librados: starting msgr at 2024-03-29T11:15:18.616 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.635+0000 7faae4451640 1 librados: starting objecter 2024-03-29T11:15:18.620 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.635+0000 7faae4451640 1 Processor -- start 2024-03-29T11:15:18.620 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.635+0000 7faae4451640 1 -- start start 2024-03-29T11:15:18.620 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.635+0000 7faae4451640 1 librados: setting wanted keys 2024-03-29T11:15:18.620 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.635+0000 7faae4451640 1 librados: calling monclient init 2024-03-29T11:15:18.621 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.635+0000 7faae4451640 10 monclient: init 2024-03-29T11:15:18.621 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.636+0000 7faae4451640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:18.621 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.636+0000 7faae4451640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:15:18.621 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.636+0000 7faae4451640 10 monclient(hunting): picked mon.noname-c con 0x7faadc0a9d70 addr v1:172.21.15.47:6790/0 2024-03-29T11:15:18.621 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.636+0000 7faae21ed640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7faadc0a9d70 0x7faadc1406a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:51914/0 (socket says 172.21.15.47:51914) 2024-03-29T11:15:18.621 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.637+0000 7faae21ed640 1 -- 172.21.15.47:0/968292927 learned_addr learned my addr 172.21.15.47:0/968292927 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:15:18.621 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.637+0000 7faae4451640 10 monclient(hunting): picked mon.noname-a con 0x7faadc13f650 addr v1:172.21.15.47:6789/0 2024-03-29T11:15:18.621 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.637+0000 7faad2ffd640 10 client.?.objecter ms_handle_connect 0x7faadc0a9d70 2024-03-29T11:15:18.621 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.637+0000 7faad2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:18.621 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.637+0000 7faae4451640 10 monclient(hunting): picked mon.noname-b con 0x7faadc1457f0 addr v1:172.21.15.121:6789/0 2024-03-29T11:15:18.621 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.637+0000 7faae4451640 1 -- 172.21.15.47:0/968292927 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faadc140210 con 0x7faadc13f650 2024-03-29T11:15:18.621 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.637+0000 7faae4451640 1 -- 172.21.15.47:0/968292927 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faadc141f60 con 0x7faadc1457f0 2024-03-29T11:15:18.621 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.638+0000 7faae4451640 1 -- 172.21.15.47:0/968292927 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faadc140db0 con 0x7faadc0a9d70 2024-03-29T11:15:18.621 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.638+0000 7faae4451640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:18.622 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.638+0000 7faae4451640 10 monclient(hunting): authenticate will time out at 961.826015s 2024-03-29T11:15:18.622 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.638+0000 7faad2ffd640 1 -- 172.21.15.47:0/968292927 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1069940975 0 0) 0x7faadc140db0 con 0x7faadc0a9d70 2024-03-29T11:15:18.622 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.638+0000 7faad2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:15:18.622 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.638+0000 7faad2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:18.622 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.638+0000 7faad2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:18.622 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.638+0000 7faad2ffd640 10 monclient(hunting): my global_id is 4331 2024-03-29T11:15:18.622 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.638+0000 7faad2ffd640 1 -- 172.21.15.47:0/968292927 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faac0004eb0 con 0x7faadc0a9d70 2024-03-29T11:15:18.622 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.638+0000 7faad2ffd640 10 client.?.objecter ms_handle_connect 0x7faadc1457f0 2024-03-29T11:15:18.625 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.638+0000 7faad2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:18.626 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.638+0000 7faad2ffd640 1 -- 172.21.15.47:0/968292927 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 597919153 0 0) 0x7faac0004eb0 con 0x7faadc0a9d70 2024-03-29T11:15:18.626 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.638+0000 7faad2ffd640 1 -- 172.21.15.47:0/968292927 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faadc140db0 con 0x7faadc0a9d70 2024-03-29T11:15:18.626 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.638+0000 7faad2ffd640 1 -- 172.21.15.47:0/968292927 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7faad8001870 con 0x7faadc0a9d70 2024-03-29T11:15:18.626 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.638+0000 7faad2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:18.626 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.638+0000 7faad2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:15:18.626 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.638+0000 7faad2ffd640 10 monclient(hunting): dump: 2024-03-29T11:15:18.626 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:15:18.626 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:15:18.626 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:18.626 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:18.626 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:18.626 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:15:18.626 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:15:18.626 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:15:18.627 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:15:18.627 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:15:18.627 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.638+0000 7faad2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:15:18.627 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.639+0000 7faad2ffd640 1 -- 172.21.15.47:0/968292927 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4245063284 0 0) 0x7faadc140db0 con 0x7faadc0a9d70 2024-03-29T11:15:18.627 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.639+0000 7faad2ffd640 1 -- 172.21.15.47:0/968292927 >> v1:172.21.15.121:6789/0 conn(0x7faadc1457f0 legacy=0x7faadc1e2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:18.627 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.639+0000 7faad2ffd640 1 -- 172.21.15.47:0/968292927 >> v1:172.21.15.47:6789/0 conn(0x7faadc13f650 legacy=0x7faadc1350d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:15:18.627 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.640+0000 7faad2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:15:18.627 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.640+0000 7faad2ffd640 1 monclient: found mon.c 2024-03-29T11:15:18.627 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.640+0000 7faad2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:18.627 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.640+0000 7faad2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:15:18.627 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.640+0000 7faad2ffd640 1 -- 172.21.15.47:0/968292927 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faadc13b800 con 0x7faadc0a9d70 2024-03-29T11:15:18.627 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.640+0000 7faad2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:15:18.627 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.640+0000 7faad2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:15:18.627 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.640+0000 7faad2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:18.627 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.640+0000 7faae4451640 5 monclient: authenticate success, global_id 4331 2024-03-29T11:15:18.628 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.640+0000 7faad2ffd640 1 -- 172.21.15.47:0/968292927 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faad80019d0 con 0x7faadc0a9d70 2024-03-29T11:15:18.628 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.640+0000 7faad2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:18.628 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.640+0000 7faad2ffd640 1 -- 172.21.15.47:0/968292927 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7faad8004a00 con 0x7faadc0a9d70 2024-03-29T11:15:18.628 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.640+0000 7faad2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:18.628 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.640+0000 7faad2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:15:18.628 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.640+0000 7faad2ffd640 10 monclient: dump: 2024-03-29T11:15:18.628 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:15:18.628 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:15:18.628 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:18.628 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:18.628 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:18.628 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:15:18.628 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:15:18.628 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:15:18.628 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:15:18.629 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:15:18.629 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.640+0000 7faae09ea640 4 set_mon_vals no callback set 2024-03-29T11:15:18.629 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.640+0000 7faae4451640 10 monclient: _renew_subs 2024-03-29T11:15:18.629 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.640+0000 7faae4451640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:15:18.629 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.640+0000 7faae4451640 1 -- 172.21.15.47:0/968292927 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faadc13fda0 con 0x7faadc0a9d70 2024-03-29T11:15:18.629 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.641+0000 7faae4451640 10 client.4331.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:15:18.629 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.641+0000 7faae4451640 10 monclient: _renew_subs 2024-03-29T11:15:18.629 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.641+0000 7faae4451640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:15:18.629 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.641+0000 7faae4451640 1 -- 172.21.15.47:0/968292927 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faadc138ee0 con 0x7faadc0a9d70 2024-03-29T11:15:18.629 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.641+0000 7faae4451640 1 librados: init done 2024-03-29T11:15:18.629 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.642+0000 7faae4451640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:18.629 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.642+0000 7faae4451640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:18.629 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.642+0000 7faae4451640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:15:18.631 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.642+0000 7faae4451640 1 -- 172.21.15.47:0/968292927 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faadc0b32c0 con 0x7faadc0a9d70 2024-03-29T11:15:18.631 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.643+0000 7faad2ffd640 1 -- 172.21.15.47:0/968292927 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7faad801a9d0 con 0x7faadc0a9d70 2024-03-29T11:15:18.631 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.644+0000 7faad2ffd640 1 -- 172.21.15.47:0/968292927 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7faad8084ad0 con 0x7faadc0a9d70 2024-03-29T11:15:18.631 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.644+0000 7faad2ffd640 10 client.4331.objecter ms_dispatch 0x7faadc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:15:18.631 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.644+0000 7faad2ffd640 3 client.4331.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:15:18.631 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.644+0000 7faad2ffd640 3 client.4331.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:15:18.631 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.644+0000 7faad2ffd640 20 client.4331.objecter dump_active .. 0 homeless 2024-03-29T11:15:18.632 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.646+0000 7faae19ec640 10 monclient: get_auth_request con 0x7faac00763f0 auth_method 0 2024-03-29T11:15:18.632 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.646+0000 7faad2ffd640 10 client.4331.objecter ms_handle_connect 0x7faac00763f0 2024-03-29T11:15:18.632 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.651+0000 7faad2ffd640 1 -- 172.21.15.47:0/968292927 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faad8086e60 con 0x7faadc0a9d70 2024-03-29T11:15:18.632 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.651+0000 7faad2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:18.632 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.651+0000 7faad2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:15:18.933 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.955+0000 7faae4451640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:18.933 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.955+0000 7faae4451640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:18.933 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.955+0000 7faae4451640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:15:18.933 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.955+0000 7faae4451640 1 -- 172.21.15.47:0/968292927 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faadc13b990 con 0x7faadc0a9d70 2024-03-29T11:15:18.933 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.956+0000 7faad2ffd640 1 -- 172.21.15.47:0/968292927 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7faadc13b990 con 0x7faadc0a9d70 2024-03-29T11:15:18.934 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.956+0000 7faad2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:18.934 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.956+0000 7faad2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:15:18.934 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:15:18.934 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:15:18.937 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.959+0000 7faae4451640 10 librados: watch_flush enter 2024-03-29T11:15:18.937 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.959+0000 7faae4451640 10 librados: watch_flush exit 2024-03-29T11:15:18.937 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.959+0000 7faae4451640 20 client.4331.objecter shutdown clearing up homeless session... 2024-03-29T11:15:18.937 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.959+0000 7faae4451640 10 client.4331.objecter successfully canceled tick 2024-03-29T11:15:18.937 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.959+0000 7faae4451640 1 -- 172.21.15.47:0/968292927 >> v1:172.21.15.121:6816/1964117370 conn(0x7faac00763f0 legacy=0x7faac00788b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:18.937 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.959+0000 7faae4451640 10 monclient: shutdown 2024-03-29T11:15:18.937 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.959+0000 7faae4451640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:18.937 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.959+0000 7faae4451640 1 -- 172.21.15.47:0/968292927 >> v1:172.21.15.47:6790/0 conn(0x7faadc0a9d70 legacy=0x7faadc1406a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:18.938 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.960+0000 7faae4451640 1 -- 172.21.15.47:0/968292927 shutdown_connections 2024-03-29T11:15:18.938 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.960+0000 7faae4451640 1 -- 172.21.15.47:0/968292927 >> 172.21.15.47:0/968292927 conn(0x7faadc066a00 msgr2=0x7faadc0b2520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:18.938 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.961+0000 7faae4451640 1 -- 172.21.15.47:0/968292927 shutdown_connections 2024-03-29T11:15:18.938 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.961+0000 7faae4451640 1 -- 172.21.15.47:0/968292927 wait complete. 2024-03-29T11:15:18.939 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:18.961+0000 7faae4451640 1 librados: shutdown 2024-03-29T11:15:18.953 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:15:25.760 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:15:25.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.929+0000 7f5a14ed3640 10 monclient: get_monmap_and_config 2024-03-29T11:15:25.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.929+0000 7f5a14ed3640 10 monclient: build_initial_monmap 2024-03-29T11:15:25.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.929+0000 7f5a14ed3640 10 monclient: monmap: 2024-03-29T11:15:25.907 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:15:25.907 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:15:25.907 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:15:25.930429+0000 2024-03-29T11:15:25.907 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:15:25.930429+0000 2024-03-29T11:15:25.907 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:25.907 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:15:25.907 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:15:25.907 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:15:25.908 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:15:25.908 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:15:25.908 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.930+0000 7f5a14ed3640 1 Processor -- start 2024-03-29T11:15:25.908 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.930+0000 7f5a14ed3640 1 -- start start 2024-03-29T11:15:25.908 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.930+0000 7f5a14ed3640 10 monclient: init 2024-03-29T11:15:25.909 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.932+0000 7f5a14ed3640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:25.909 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.932+0000 7f5a14ed3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:15:25.909 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.932+0000 7f5a14ed3640 10 monclient(hunting): picked mon.noname-c con 0x7f5a100b0860 addr v1:172.21.15.121:6789/0 2024-03-29T11:15:25.909 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.932+0000 7f5a14ed3640 10 monclient(hunting): picked mon.noname-b con 0x7f5a1013ef30 addr v1:172.21.15.47:6790/0 2024-03-29T11:15:25.909 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.932+0000 7f5a14ed3640 10 monclient(hunting): picked mon.noname-a con 0x7f5a100af8e0 addr v1:172.21.15.47:6789/0 2024-03-29T11:15:25.909 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.932+0000 7f5a14ed3640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a100b0c40 con 0x7f5a100af8e0 2024-03-29T11:15:25.909 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.932+0000 7f5a14ed3640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a100aed10 con 0x7f5a100b0860 2024-03-29T11:15:25.909 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.932+0000 7f5a14ed3640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a100afd50 con 0x7f5a1013ef30 2024-03-29T11:15:25.910 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.932+0000 7f5a14ed3640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:25.910 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.932+0000 7f5a14ed3640 10 monclient(hunting): authenticate will time out at 969.120122s 2024-03-29T11:15:25.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.934+0000 7f5a0dd9b640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f5a1013ef30 0x7f5a100af1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:50122/0 (socket says 172.21.15.47:50122) 2024-03-29T11:15:25.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.934+0000 7f5a0dd9b640 1 -- 172.21.15.47:0/1355349929 learned_addr learned my addr 172.21.15.47:0/1355349929 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:15:25.914 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.936+0000 7f5a0d59a640 1 -- 172.21.15.47:0/1355349929 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 890437734 0 0) 0x7f5a100afd50 con 0x7f5a1013ef30 2024-03-29T11:15:25.914 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.936+0000 7f5a0d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:15:25.914 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.936+0000 7f5a0d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:25.914 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.936+0000 7f5a0d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:25.914 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.936+0000 7f5a0d59a640 10 monclient(hunting): my global_id is 4334 2024-03-29T11:15:25.914 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.936+0000 7f5a0d59a640 1 -- 172.21.15.47:0/1355349929 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59f8003520 con 0x7f5a1013ef30 2024-03-29T11:15:25.914 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.936+0000 7f5a0d59a640 1 -- 172.21.15.47:0/1355349929 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4225121048 0 0) 0x7f5a100b0c40 con 0x7f5a100af8e0 2024-03-29T11:15:25.915 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.936+0000 7f5a0d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:15:25.915 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.936+0000 7f5a0d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:25.915 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.936+0000 7f5a0d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:25.915 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.937+0000 7f5a0d59a640 10 monclient(hunting): my global_id is 4311 2024-03-29T11:15:25.915 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.937+0000 7f5a0d59a640 1 -- 172.21.15.47:0/1355349929 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a100afd50 con 0x7f5a100af8e0 2024-03-29T11:15:25.915 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.937+0000 7f5a0d59a640 1 -- 172.21.15.47:0/1355349929 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2315270496 0 0) 0x7f59f8003520 con 0x7f5a1013ef30 2024-03-29T11:15:25.915 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.937+0000 7f5a0d59a640 1 -- 172.21.15.47:0/1355349929 >> v1:172.21.15.121:6789/0 conn(0x7f5a100b0860 legacy=0x7f5a1013e820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:15:25.919 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.941+0000 7f5a0d59a640 1 -- 172.21.15.47:0/1355349929 >> v1:172.21.15.47:6789/0 conn(0x7f5a100af8e0 legacy=0x7f5a10149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:25.919 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.941+0000 7f5a0d59a640 10 monclient: _finish_hunting 0 2024-03-29T11:15:25.919 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.941+0000 7f5a0d59a640 1 monclient: found mon.noname-b 2024-03-29T11:15:25.919 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.941+0000 7f5a0d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:25.919 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.941+0000 7f5a0d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-29T11:15:25.919 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.941+0000 7f5a0d59a640 1 -- 172.21.15.47:0/1355349929 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a100ab1b0 con 0x7f5a1013ef30 2024-03-29T11:15:25.919 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.941+0000 7f5a0d59a640 10 monclient: _finish_auth 0 2024-03-29T11:15:25.919 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.941+0000 7f5a0d59a640 10 monclient: _check_auth_tickets 2024-03-29T11:15:25.920 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.941+0000 7f5a0d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:25.920 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.941+0000 7f5a0d59a640 1 -- 172.21.15.47:0/1355349929 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f5a08001a10 con 0x7f5a1013ef30 2024-03-29T11:15:25.920 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.941+0000 7f5a0d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:25.920 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.941+0000 7f5a0d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:15:25.920 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.941+0000 7f5a0d59a640 10 monclient: dump: 2024-03-29T11:15:25.920 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:15:25.920 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:15:25.920 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:25.920 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:25.920 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:25.920 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:15:25.920 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:15:25.920 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:15:25.920 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:15:25.920 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:15:25.921 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.941+0000 7f5a14ed3640 5 monclient: authenticate success, global_id 4334 2024-03-29T11:15:25.921 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.942+0000 7f5a14ed3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:15:25.921 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.942+0000 7f5a0d59a640 1 -- 172.21.15.47:0/1355349929 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a08001b70 con 0x7f5a1013ef30 2024-03-29T11:15:25.921 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.942+0000 7f5a0d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:25.921 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.942+0000 7f5a0d59a640 1 -- 172.21.15.47:0/1355349929 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f5a08001fc0 con 0x7f5a1013ef30 2024-03-29T11:15:25.921 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.942+0000 7f5a0d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:25.921 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.942+0000 7f5a0d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:15:25.922 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.942+0000 7f5a0d59a640 10 monclient: dump: 2024-03-29T11:15:25.922 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:15:25.922 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:15:25.922 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:25.922 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:25.922 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:25.922 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:15:25.922 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:15:25.922 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:15:25.922 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:15:25.922 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:15:25.922 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.942+0000 7f5a14ed3640 10 monclient: get_monmap_and_config success 2024-03-29T11:15:25.922 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.942+0000 7f5a14ed3640 4 set_mon_vals no callback set 2024-03-29T11:15:25.923 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.943+0000 7f5a14ed3640 10 monclient: shutdown 2024-03-29T11:15:25.923 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.943+0000 7f5a14ed3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:25.923 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.943+0000 7f5a14ed3640 1 -- 172.21.15.47:0/1355349929 >> v1:172.21.15.47:6790/0 conn(0x7f5a1013ef30 legacy=0x7f5a100af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:25.923 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.945+0000 7f5a14ed3640 1 -- 172.21.15.47:0/1355349929 shutdown_connections 2024-03-29T11:15:25.923 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.945+0000 7f5a14ed3640 1 -- 172.21.15.47:0/1355349929 >> 172.21.15.47:0/1355349929 conn(0x7f5a1006b660 msgr2=0x7f5a1006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:25.925 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.947+0000 7f5a14ed3640 1 -- 172.21.15.47:0/1355349929 shutdown_connections 2024-03-29T11:15:25.925 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.948+0000 7f5a14ed3640 1 -- 172.21.15.47:0/1355349929 wait complete. 2024-03-29T11:15:25.926 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.949+0000 7f5a14ed3640 10 monclient: build_initial_monmap 2024-03-29T11:15:25.926 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.949+0000 7f5a14ed3640 10 monclient: monmap: 2024-03-29T11:15:25.926 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:15:25.927 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:15:25.927 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:15:25.927 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:15:25.927 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:25.927 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:15:25.927 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:15:25.927 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:15:25.927 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:15:25.927 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:15:25.929 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.949+0000 7f5a14ed3640 1 librados: starting msgr at 2024-03-29T11:15:25.930 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.949+0000 7f5a14ed3640 1 librados: starting objecter 2024-03-29T11:15:25.930 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.950+0000 7f5a14ed3640 1 Processor -- start 2024-03-29T11:15:25.930 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.950+0000 7f5a14ed3640 1 -- start start 2024-03-29T11:15:25.930 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.950+0000 7f5a14ed3640 1 librados: setting wanted keys 2024-03-29T11:15:25.930 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.950+0000 7f5a14ed3640 1 librados: calling monclient init 2024-03-29T11:15:25.930 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.950+0000 7f5a14ed3640 10 monclient: init 2024-03-29T11:15:25.930 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.951+0000 7f5a14ed3640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:25.930 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.951+0000 7f5a14ed3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:15:25.930 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.951+0000 7f5a14ed3640 10 monclient(hunting): picked mon.noname-a con 0x7f5a100af8e0 addr v1:172.21.15.47:6789/0 2024-03-29T11:15:25.930 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.951+0000 7f5a14ed3640 10 monclient(hunting): picked mon.noname-b con 0x7f5a100b0860 addr v1:172.21.15.121:6789/0 2024-03-29T11:15:25.930 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.951+0000 7f5a14ed3640 10 monclient(hunting): picked mon.noname-c con 0x7f5a1013ef30 addr v1:172.21.15.47:6790/0 2024-03-29T11:15:25.930 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.951+0000 7f5a14ed3640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a100b1440 con 0x7f5a100af8e0 2024-03-29T11:15:25.931 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.951+0000 7f5a14ed3640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a100b1e70 con 0x7f5a100b0860 2024-03-29T11:15:25.931 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.951+0000 7f5a14ed3640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a100b2150 con 0x7f5a1013ef30 2024-03-29T11:15:25.931 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.951+0000 7f5a14ed3640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:25.931 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.951+0000 7f5a14ed3640 10 monclient(hunting): authenticate will time out at 969.139400s 2024-03-29T11:15:25.931 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.951+0000 7f5a0ed9d640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f5a1013ef30 0x7f5a1012df70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:58548/0 (socket says 172.21.15.47:58548) 2024-03-29T11:15:25.931 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.951+0000 7f5a0ed9d640 1 -- 172.21.15.47:0/3306416344 learned_addr learned my addr 172.21.15.47:0/3306416344 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:15:25.932 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.954+0000 7f59feffd640 10 client.?.objecter ms_handle_connect 0x7f5a1013ef30 2024-03-29T11:15:25.932 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.955+0000 7f59feffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:25.933 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.955+0000 7f59feffd640 1 -- 172.21.15.47:0/3306416344 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3374786067 0 0) 0x7f5a100b2150 con 0x7f5a1013ef30 2024-03-29T11:15:25.933 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.955+0000 7f59feffd640 10 monclient(hunting): _negotiate 2024-03-29T11:15:25.933 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.955+0000 7f59feffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:25.933 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.955+0000 7f59feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:25.933 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.955+0000 7f59feffd640 10 monclient(hunting): my global_id is 4337 2024-03-29T11:15:25.933 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.955+0000 7f59feffd640 1 -- 172.21.15.47:0/3306416344 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59f0004eb0 con 0x7f5a1013ef30 2024-03-29T11:15:25.936 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.955+0000 7f59feffd640 1 -- 172.21.15.47:0/3306416344 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2152836893 0 0) 0x7f59f0004eb0 con 0x7f5a1013ef30 2024-03-29T11:15:25.936 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.956+0000 7f59feffd640 1 -- 172.21.15.47:0/3306416344 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a100b2150 con 0x7f5a1013ef30 2024-03-29T11:15:25.936 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.956+0000 7f59feffd640 1 -- 172.21.15.47:0/3306416344 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f5a04001900 con 0x7f5a1013ef30 2024-03-29T11:15:25.936 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.956+0000 7f59feffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:25.936 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.956+0000 7f59feffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:15:25.936 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.956+0000 7f59feffd640 10 monclient(hunting): dump: 2024-03-29T11:15:25.937 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:15:25.937 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:15:25.937 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:25.937 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:25.937 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:25.937 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:15:25.937 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:15:25.937 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:15:25.937 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:15:25.937 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:15:25.937 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.956+0000 7f59feffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:15:25.937 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.956+0000 7f59feffd640 1 -- 172.21.15.47:0/3306416344 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2944404996 0 0) 0x7f5a100b2150 con 0x7f5a1013ef30 2024-03-29T11:15:25.937 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.956+0000 7f59feffd640 1 -- 172.21.15.47:0/3306416344 >> v1:172.21.15.121:6789/0 conn(0x7f5a100b0860 legacy=0x7f5a1012d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:25.937 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.956+0000 7f59feffd640 1 -- 172.21.15.47:0/3306416344 >> v1:172.21.15.47:6789/0 conn(0x7f5a100af8e0 legacy=0x7f5a1013a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:25.938 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.956+0000 7f59feffd640 10 monclient: _finish_hunting 0 2024-03-29T11:15:25.938 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.956+0000 7f59feffd640 1 monclient: found mon.c 2024-03-29T11:15:25.938 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.957+0000 7f59feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:25.938 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.957+0000 7f59feffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:15:25.938 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.957+0000 7f59feffd640 1 -- 172.21.15.47:0/3306416344 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a100b2430 con 0x7f5a1013ef30 2024-03-29T11:15:25.938 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.957+0000 7f59feffd640 10 monclient: _finish_auth 0 2024-03-29T11:15:25.938 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.957+0000 7f59feffd640 10 monclient: _check_auth_tickets 2024-03-29T11:15:25.938 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.957+0000 7f59feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:25.938 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.958+0000 7f59feffd640 10 client.?.objecter ms_handle_connect 0x7f5a100b0860 2024-03-29T11:15:25.938 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.958+0000 7f5a14ed3640 5 monclient: authenticate success, global_id 4337 2024-03-29T11:15:25.938 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.958+0000 7f5a14ed3640 10 monclient: _renew_subs 2024-03-29T11:15:25.938 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.958+0000 7f5a14ed3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:15:25.938 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.958+0000 7f5a14ed3640 1 -- 172.21.15.47:0/3306416344 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a101e2c40 con 0x7f5a1013ef30 2024-03-29T11:15:25.939 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.958+0000 7f5a14ed3640 10 client.4337.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:15:25.939 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.958+0000 7f5a14ed3640 10 monclient: _renew_subs 2024-03-29T11:15:25.939 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.958+0000 7f5a14ed3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:15:25.939 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.958+0000 7f5a14ed3640 1 -- 172.21.15.47:0/3306416344 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a101e2fb0 con 0x7f5a1013ef30 2024-03-29T11:15:25.939 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.958+0000 7f5a14ed3640 1 librados: init done 2024-03-29T11:15:25.941 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.960+0000 7f59feffd640 10 client.4337.objecter resend_mon_ops 2024-03-29T11:15:25.942 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.960+0000 7f59feffd640 1 -- 172.21.15.47:0/3306416344 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a04001a60 con 0x7f5a1013ef30 2024-03-29T11:15:25.942 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.960+0000 7f59feffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:25.942 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.960+0000 7f59fffff640 4 set_mon_vals no callback set 2024-03-29T11:15:25.942 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.960+0000 7f59feffd640 1 -- 172.21.15.47:0/3306416344 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f5a040038a0 con 0x7f5a1013ef30 2024-03-29T11:15:25.942 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.960+0000 7f59feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:25.942 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.960+0000 7f59feffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:15:25.942 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.960+0000 7f59feffd640 10 monclient: dump: 2024-03-29T11:15:25.942 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:15:25.942 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:15:25.942 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:25.942 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:25.942 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:25.942 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:15:25.943 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:15:25.943 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:15:25.943 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:15:25.943 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:15:25.943 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.961+0000 7f59feffd640 1 -- 172.21.15.47:0/3306416344 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7f5a04003b00 con 0x7f5a1013ef30 2024-03-29T11:15:25.943 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.962+0000 7f59fcff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:25.943 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.962+0000 7f59fcff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:25.943 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.962+0000 7f59fcff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:15:25.943 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.962+0000 7f59fcff9640 1 -- 172.21.15.47:0/3306416344 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59e8005080 con 0x7f5a1013ef30 2024-03-29T11:15:25.943 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.963+0000 7f59feffd640 1 -- 172.21.15.47:0/3306416344 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7f5a04084ae0 con 0x7f5a1013ef30 2024-03-29T11:15:25.943 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.963+0000 7f59feffd640 10 client.4337.objecter ms_dispatch 0x7f5a10000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:15:25.943 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.963+0000 7f59feffd640 3 client.4337.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:15:25.943 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.963+0000 7f59feffd640 3 client.4337.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:15:25.943 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.963+0000 7f59feffd640 20 client.4337.objecter dump_active .. 0 homeless 2024-03-29T11:15:25.944 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.965+0000 7f5a0e59c640 10 monclient: get_auth_request con 0x7f59f006dc60 auth_method 0 2024-03-29T11:15:25.945 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.968+0000 7f59feffd640 10 client.4337.objecter ms_handle_connect 0x7f59f006dc60 2024-03-29T11:15:25.947 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.970+0000 7f59feffd640 1 -- 172.21.15.47:0/3306416344 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5a040504f0 con 0x7f5a1013ef30 2024-03-29T11:15:25.947 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.970+0000 7f59feffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:25.948 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:25.970+0000 7f59feffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:15:26.287 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:26.310+0000 7f59fcff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:26.288 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:26.310+0000 7f59fcff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:26.288 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:26.310+0000 7f59fcff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:15:26.288 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:26.310+0000 7f59fcff9640 1 -- 172.21.15.47:0/3306416344 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f59e8005330 con 0x7f5a1013ef30 2024-03-29T11:15:26.289 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:26.312+0000 7f59feffd640 1 -- 172.21.15.47:0/3306416344 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7f5a04052420 con 0x7f5a1013ef30 2024-03-29T11:15:26.289 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:26.312+0000 7f59feffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:26.290 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:26.312+0000 7f59feffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:15:26.290 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:15:26.290 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:15:26.292 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:26.315+0000 7f59fcff9640 10 librados: watch_flush enter 2024-03-29T11:15:26.293 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:26.315+0000 7f59fcff9640 10 librados: watch_flush exit 2024-03-29T11:15:26.293 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:26.315+0000 7f59fcff9640 20 client.4337.objecter shutdown clearing up homeless session... 2024-03-29T11:15:26.293 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:26.316+0000 7f59fcff9640 10 client.4337.objecter successfully canceled tick 2024-03-29T11:15:26.294 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:26.316+0000 7f59fcff9640 1 -- 172.21.15.47:0/3306416344 >> v1:172.21.15.121:6816/1964117370 conn(0x7f59f006dc60 legacy=0x7f59f0070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:26.294 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:26.316+0000 7f59fcff9640 10 monclient: shutdown 2024-03-29T11:15:26.294 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:26.316+0000 7f59fcff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:26.294 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:26.316+0000 7f59fcff9640 1 -- 172.21.15.47:0/3306416344 >> v1:172.21.15.47:6790/0 conn(0x7f5a1013ef30 legacy=0x7f5a1012df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:26.294 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:26.316+0000 7f59fcff9640 1 -- 172.21.15.47:0/3306416344 shutdown_connections 2024-03-29T11:15:26.294 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:26.316+0000 7f59fcff9640 1 -- 172.21.15.47:0/3306416344 >> 172.21.15.47:0/3306416344 conn(0x7f5a1006b660 msgr2=0x7f5a1013e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:26.294 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:26.316+0000 7f59fcff9640 1 -- 172.21.15.47:0/3306416344 shutdown_connections 2024-03-29T11:15:26.294 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:26.316+0000 7f59fcff9640 1 -- 172.21.15.47:0/3306416344 wait complete. 2024-03-29T11:15:26.294 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:26.316+0000 7f59fcff9640 1 librados: shutdown 2024-03-29T11:15:26.312 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:15:33.120 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:15:33.271 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.294+0000 7f0c0de4d640 10 monclient: get_monmap_and_config 2024-03-29T11:15:33.271 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.294+0000 7f0c0de4d640 10 monclient: build_initial_monmap 2024-03-29T11:15:33.272 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.294+0000 7f0c0de4d640 10 monclient: monmap: 2024-03-29T11:15:33.272 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:15:33.272 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:15:33.272 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:15:33.295111+0000 2024-03-29T11:15:33.272 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:15:33.295111+0000 2024-03-29T11:15:33.272 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:33.272 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:15:33.272 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:15:33.272 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:15:33.272 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:15:33.272 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:15:33.273 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.295+0000 7f0c0de4d640 1 Processor -- start 2024-03-29T11:15:33.273 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.295+0000 7f0c0de4d640 1 -- start start 2024-03-29T11:15:33.273 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.295+0000 7f0c0de4d640 10 monclient: init 2024-03-29T11:15:33.280 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.296+0000 7f0c0de4d640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:33.280 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.296+0000 7f0c0de4d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:15:33.280 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.296+0000 7f0c0de4d640 10 monclient(hunting): picked mon.noname-b con 0x7f0c080b1f60 addr v1:172.21.15.47:6790/0 2024-03-29T11:15:33.280 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.296+0000 7f0c0de4d640 10 monclient(hunting): picked mon.noname-c con 0x7f0c08071eb0 addr v1:172.21.15.121:6789/0 2024-03-29T11:15:33.281 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.296+0000 7f0c0de4d640 10 monclient(hunting): picked mon.noname-a con 0x7f0c08143d10 addr v1:172.21.15.47:6789/0 2024-03-29T11:15:33.281 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.296+0000 7f0c0de4d640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c0806ef80 con 0x7f0c08143d10 2024-03-29T11:15:33.281 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.296+0000 7f0c0de4d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c0806f260 con 0x7f0c08071eb0 2024-03-29T11:15:33.281 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.296+0000 7f0c0de4d640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c080b40c0 con 0x7f0c080b1f60 2024-03-29T11:15:33.281 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.296+0000 7f0c0de4d640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:33.281 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.296+0000 7f0c0de4d640 10 monclient(hunting): authenticate will time out at 976.484812s 2024-03-29T11:15:33.281 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.299+0000 7f0c07fff640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f0c08143d10 0x7f0c08146b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:37571/0 (socket says 172.21.15.47:37571) 2024-03-29T11:15:33.281 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.300+0000 7f0c07fff640 1 -- 172.21.15.47:0/3442567786 learned_addr learned my addr 172.21.15.47:0/3442567786 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:15:33.281 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.300+0000 7f0c067fc640 1 -- 172.21.15.47:0/3442567786 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2074396534 0 0) 0x7f0c0806ef80 con 0x7f0c08143d10 2024-03-29T11:15:33.281 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.300+0000 7f0c067fc640 10 monclient(hunting): _negotiate 2024-03-29T11:15:33.281 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.300+0000 7f0c067fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:33.281 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.300+0000 7f0c067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:33.282 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.300+0000 7f0c067fc640 10 monclient(hunting): my global_id is 4314 2024-03-29T11:15:33.282 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.300+0000 7f0c067fc640 1 -- 172.21.15.47:0/3442567786 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bf0003520 con 0x7f0c08143d10 2024-03-29T11:15:33.282 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.300+0000 7f0c067fc640 1 -- 172.21.15.47:0/3442567786 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3387308740 0 0) 0x7f0bf0003520 con 0x7f0c08143d10 2024-03-29T11:15:33.282 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.300+0000 7f0c067fc640 1 -- 172.21.15.47:0/3442567786 >> v1:172.21.15.47:6790/0 conn(0x7f0c080b1f60 legacy=0x7f0c08143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:15:33.282 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.300+0000 7f0c067fc640 1 -- 172.21.15.47:0/3442567786 >> v1:172.21.15.121:6789/0 conn(0x7f0c08071eb0 legacy=0x7f0c08146470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:15:33.282 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.301+0000 7f0c067fc640 10 monclient: _finish_hunting 0 2024-03-29T11:15:33.282 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.301+0000 7f0c067fc640 1 monclient: found mon.noname-a 2024-03-29T11:15:33.282 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.301+0000 7f0c067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:33.282 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.301+0000 7f0c067fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-29T11:15:33.282 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.301+0000 7f0c067fc640 1 -- 172.21.15.47:0/3442567786 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c08155d70 con 0x7f0c08143d10 2024-03-29T11:15:33.282 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.301+0000 7f0c067fc640 10 monclient: _finish_auth 0 2024-03-29T11:15:33.282 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.301+0000 7f0c067fc640 10 monclient: _check_auth_tickets 2024-03-29T11:15:33.282 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.301+0000 7f0c067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:33.282 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.301+0000 7f0c067fc640 1 -- 172.21.15.47:0/3442567786 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f0c00001a10 con 0x7f0c08143d10 2024-03-29T11:15:33.282 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.301+0000 7f0c067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:33.283 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.301+0000 7f0c067fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:15:33.283 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.301+0000 7f0c067fc640 10 monclient: dump: 2024-03-29T11:15:33.283 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:15:33.283 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:15:33.283 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:33.283 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:33.283 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:33.283 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:15:33.283 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:15:33.283 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:15:33.283 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:15:33.283 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:15:33.283 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.303+0000 7f0c0de4d640 5 monclient: authenticate success, global_id 4314 2024-03-29T11:15:33.283 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.303+0000 7f0c0de4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:15:33.284 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.304+0000 7f0c067fc640 1 -- 172.21.15.47:0/3442567786 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0c00001ca0 con 0x7f0c08143d10 2024-03-29T11:15:33.284 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.304+0000 7f0c067fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:33.284 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.304+0000 7f0c067fc640 1 -- 172.21.15.47:0/3442567786 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f0c00002030 con 0x7f0c08143d10 2024-03-29T11:15:33.284 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.304+0000 7f0c067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:33.284 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.304+0000 7f0c067fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:15:33.284 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.304+0000 7f0c067fc640 10 monclient: dump: 2024-03-29T11:15:33.284 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:15:33.284 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:15:33.284 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:33.285 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:33.285 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:33.285 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:15:33.285 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:15:33.285 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:15:33.285 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:15:33.285 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:15:33.285 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.306+0000 7f0c0de4d640 10 monclient: get_monmap_and_config success 2024-03-29T11:15:33.286 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.306+0000 7f0c0de4d640 4 set_mon_vals no callback set 2024-03-29T11:15:33.286 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.307+0000 7f0c0de4d640 10 monclient: shutdown 2024-03-29T11:15:33.286 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.307+0000 7f0c0de4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:33.286 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.307+0000 7f0c0de4d640 1 -- 172.21.15.47:0/3442567786 >> v1:172.21.15.47:6789/0 conn(0x7f0c08143d10 legacy=0x7f0c08146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:33.286 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.307+0000 7f0c0de4d640 1 -- 172.21.15.47:0/3442567786 shutdown_connections 2024-03-29T11:15:33.286 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.307+0000 7f0c0de4d640 1 -- 172.21.15.47:0/3442567786 >> 172.21.15.47:0/3442567786 conn(0x7f0c08066a00 msgr2=0x7f0c08066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:33.287 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.308+0000 7f0c0de4d640 1 -- 172.21.15.47:0/3442567786 shutdown_connections 2024-03-29T11:15:33.287 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.308+0000 7f0c0de4d640 1 -- 172.21.15.47:0/3442567786 wait complete. 2024-03-29T11:15:33.287 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.309+0000 7f0c0de4d640 10 monclient: build_initial_monmap 2024-03-29T11:15:33.287 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.309+0000 7f0c0de4d640 10 monclient: monmap: 2024-03-29T11:15:33.287 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:15:33.287 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:15:33.287 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:15:33.287 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:15:33.287 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:33.287 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:15:33.287 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:15:33.288 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:15:33.288 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:15:33.288 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:15:33.288 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.309+0000 7f0c0de4d640 1 librados: starting msgr at 2024-03-29T11:15:33.288 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.309+0000 7f0c0de4d640 1 librados: starting objecter 2024-03-29T11:15:33.288 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.309+0000 7f0c0de4d640 1 Processor -- start 2024-03-29T11:15:33.290 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.313+0000 7f0c0de4d640 1 -- start start 2024-03-29T11:15:33.290 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.313+0000 7f0c0de4d640 1 librados: setting wanted keys 2024-03-29T11:15:33.290 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.313+0000 7f0c0de4d640 1 librados: calling monclient init 2024-03-29T11:15:33.291 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.313+0000 7f0c0de4d640 10 monclient: init 2024-03-29T11:15:33.297 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.314+0000 7f0c0de4d640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:33.297 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.314+0000 7f0c0de4d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:15:33.297 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.314+0000 7f0c0de4d640 10 monclient(hunting): picked mon.noname-b con 0x7f0c080b1f60 addr v1:172.21.15.121:6789/0 2024-03-29T11:15:33.297 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.317+0000 7f0c0de4d640 10 monclient(hunting): picked mon.noname-c con 0x7f0c08143d10 addr v1:172.21.15.47:6790/0 2024-03-29T11:15:33.297 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.317+0000 7f0c0de4d640 10 monclient(hunting): picked mon.noname-a con 0x7f0c08071410 addr v1:172.21.15.47:6789/0 2024-03-29T11:15:33.298 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.317+0000 7f0c0de4d640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c080705e0 con 0x7f0c08071410 2024-03-29T11:15:33.298 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.317+0000 7f0c0de4d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c08070f30 con 0x7f0c080b1f60 2024-03-29T11:15:33.298 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.317+0000 7f0c0de4d640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c0812fb40 con 0x7f0c08143d10 2024-03-29T11:15:33.298 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.317+0000 7f0c0de4d640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:33.298 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.317+0000 7f0c0de4d640 10 monclient(hunting): authenticate will time out at 976.505001s 2024-03-29T11:15:33.298 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.317+0000 7f0c06ffd640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f0c08143d10 0x7f0c0812ed20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:44465/0 (socket says 172.21.15.47:44465) 2024-03-29T11:15:33.298 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.317+0000 7f0c06ffd640 1 -- 172.21.15.47:0/3795102741 learned_addr learned my addr 172.21.15.47:0/3795102741 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:15:33.298 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.319+0000 7f0beffff640 10 client.?.objecter ms_handle_connect 0x7f0c08143d10 2024-03-29T11:15:33.298 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.320+0000 7f0beffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:33.299 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.320+0000 7f0beffff640 1 -- 172.21.15.47:0/3795102741 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2193562210 0 0) 0x7f0c0812fb40 con 0x7f0c08143d10 2024-03-29T11:15:33.304 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.320+0000 7f0beffff640 10 monclient(hunting): _negotiate 2024-03-29T11:15:33.305 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.320+0000 7f0beffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:33.305 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.320+0000 7f0beffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:33.305 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.320+0000 7f0beffff640 10 monclient(hunting): my global_id is 4340 2024-03-29T11:15:33.305 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.320+0000 7f0beffff640 1 -- 172.21.15.47:0/3795102741 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bf4004eb0 con 0x7f0c08143d10 2024-03-29T11:15:33.305 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.320+0000 7f0beffff640 10 client.?.objecter ms_handle_connect 0x7f0c080b1f60 2024-03-29T11:15:33.305 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.320+0000 7f0beffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:33.305 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.321+0000 7f0beffff640 1 -- 172.21.15.47:0/3795102741 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1458232196 0 0) 0x7f0bf4004eb0 con 0x7f0c08143d10 2024-03-29T11:15:33.305 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.321+0000 7f0beffff640 1 -- 172.21.15.47:0/3795102741 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0c0812fb40 con 0x7f0c08143d10 2024-03-29T11:15:33.305 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.321+0000 7f0beffff640 1 -- 172.21.15.47:0/3795102741 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f0bf8001870 con 0x7f0c08143d10 2024-03-29T11:15:33.305 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.321+0000 7f0beffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:33.305 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.321+0000 7f0beffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:15:33.305 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.321+0000 7f0beffff640 10 monclient(hunting): dump: 2024-03-29T11:15:33.306 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:15:33.306 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:15:33.306 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:33.306 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:33.306 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:33.306 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:15:33.306 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:15:33.306 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:15:33.306 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:15:33.306 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:15:33.306 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.321+0000 7f0beffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:15:33.306 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.324+0000 7f0beffff640 10 client.?.objecter ms_handle_connect 0x7f0c08071410 2024-03-29T11:15:33.306 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.324+0000 7f0beffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:33.307 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.324+0000 7f0beffff640 1 -- 172.21.15.47:0/3795102741 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1524684258 0 0) 0x7f0c08070f30 con 0x7f0c080b1f60 2024-03-29T11:15:33.307 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.324+0000 7f0beffff640 10 monclient(hunting): _negotiate 2024-03-29T11:15:33.307 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.324+0000 7f0beffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:33.307 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.324+0000 7f0beffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:33.307 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.324+0000 7f0beffff640 10 monclient(hunting): my global_id is 4327 2024-03-29T11:15:33.307 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.324+0000 7f0beffff640 1 -- 172.21.15.47:0/3795102741 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bf4004eb0 con 0x7f0c080b1f60 2024-03-29T11:15:33.307 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.324+0000 7f0beffff640 1 -- 172.21.15.47:0/3795102741 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3723770952 0 0) 0x7f0c0812fb40 con 0x7f0c08143d10 2024-03-29T11:15:33.307 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.324+0000 7f0beffff640 1 -- 172.21.15.47:0/3795102741 >> v1:172.21.15.121:6789/0 conn(0x7f0c080b1f60 legacy=0x7f0c081545c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:33.307 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.324+0000 7f0beffff640 1 -- 172.21.15.47:0/3795102741 >> v1:172.21.15.47:6789/0 conn(0x7f0c08071410 legacy=0x7f0c0812f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:33.307 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.324+0000 7f0beffff640 10 monclient: _finish_hunting 0 2024-03-29T11:15:33.307 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.324+0000 7f0beffff640 1 monclient: found mon.c 2024-03-29T11:15:33.307 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.324+0000 7f0beffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:33.307 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.324+0000 7f0beffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:15:33.307 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.324+0000 7f0beffff640 1 -- 172.21.15.47:0/3795102741 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c08070830 con 0x7f0c08143d10 2024-03-29T11:15:33.307 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.324+0000 7f0beffff640 10 monclient: _finish_auth 0 2024-03-29T11:15:33.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.324+0000 7f0beffff640 10 monclient: _check_auth_tickets 2024-03-29T11:15:33.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.324+0000 7f0beffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:33.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.324+0000 7f0c0de4d640 5 monclient: authenticate success, global_id 4340 2024-03-29T11:15:33.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.324+0000 7f0c0de4d640 10 monclient: _renew_subs 2024-03-29T11:15:33.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.324+0000 7f0c0de4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:15:33.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.324+0000 7f0c0de4d640 1 -- 172.21.15.47:0/3795102741 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c08070b10 con 0x7f0c08143d10 2024-03-29T11:15:33.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.325+0000 7f0c0de4d640 10 client.4340.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:15:33.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.325+0000 7f0c0de4d640 10 monclient: _renew_subs 2024-03-29T11:15:33.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.325+0000 7f0c0de4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:15:33.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.325+0000 7f0c0de4d640 1 -- 172.21.15.47:0/3795102741 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c08141d80 con 0x7f0c08143d10 2024-03-29T11:15:33.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.325+0000 7f0c0de4d640 1 librados: init done 2024-03-29T11:15:33.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.327+0000 7f0c0de4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:33.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.327+0000 7f0c0de4d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:33.309 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.327+0000 7f0c0de4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:15:33.309 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.327+0000 7f0c0de4d640 1 -- 172.21.15.47:0/3795102741 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c08075f10 con 0x7f0c08143d10 2024-03-29T11:15:33.310 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.327+0000 7f0beffff640 1 -- 172.21.15.47:0/3795102741 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0bf80019d0 con 0x7f0c08143d10 2024-03-29T11:15:33.310 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.330+0000 7f0beffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:33.310 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.330+0000 7f0beffff640 1 -- 172.21.15.47:0/3795102741 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f0bf8004aa0 con 0x7f0c08143d10 2024-03-29T11:15:33.310 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.330+0000 7f0beffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:33.310 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.330+0000 7f0beffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:15:33.310 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.330+0000 7f0beffff640 10 monclient: dump: 2024-03-29T11:15:33.310 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:15:33.310 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:15:33.310 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:33.310 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:33.311 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:33.311 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:15:33.311 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:15:33.311 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:15:33.311 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:15:33.311 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:15:33.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.330+0000 7f0c05ffb640 4 set_mon_vals no callback set 2024-03-29T11:15:33.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.331+0000 7f0beffff640 1 -- 172.21.15.47:0/3795102741 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7f0bf8004c40 con 0x7f0c08143d10 2024-03-29T11:15:33.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.331+0000 7f0beffff640 1 -- 172.21.15.47:0/3795102741 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7f0bf8084a80 con 0x7f0c08143d10 2024-03-29T11:15:33.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.331+0000 7f0beffff640 10 client.4340.objecter ms_dispatch 0x7f0c08000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:15:33.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.331+0000 7f0beffff640 3 client.4340.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:15:33.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.331+0000 7f0beffff640 3 client.4340.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:15:33.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.331+0000 7f0beffff640 20 client.4340.objecter dump_active .. 0 homeless 2024-03-29T11:15:33.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.332+0000 7f0c077fe640 10 monclient: get_auth_request con 0x7f0bf406dcc0 auth_method 0 2024-03-29T11:15:33.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.332+0000 7f0beffff640 10 client.4340.objecter ms_handle_connect 0x7f0bf406dcc0 2024-03-29T11:15:33.314 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.336+0000 7f0beffff640 1 -- 172.21.15.47:0/3795102741 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0bf8050360 con 0x7f0c08143d10 2024-03-29T11:15:33.314 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.336+0000 7f0beffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:33.314 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.336+0000 7f0beffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:15:33.647 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.670+0000 7f0c0de4d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:33.648 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.670+0000 7f0c0de4d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:33.648 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.670+0000 7f0c0de4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:15:33.648 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.670+0000 7f0c0de4d640 1 -- 172.21.15.47:0/3795102741 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0c080b3560 con 0x7f0c08143d10 2024-03-29T11:15:33.653 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.676+0000 7f0beffff640 1 -- 172.21.15.47:0/3795102741 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7f0bf8052290 con 0x7f0c08143d10 2024-03-29T11:15:33.654 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.676+0000 7f0beffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:33.654 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.676+0000 7f0beffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:15:33.654 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:15:33.654 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:15:33.657 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.680+0000 7f0c0de4d640 10 librados: watch_flush enter 2024-03-29T11:15:33.657 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.680+0000 7f0c0de4d640 10 librados: watch_flush exit 2024-03-29T11:15:33.657 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.680+0000 7f0c0de4d640 20 client.4340.objecter shutdown clearing up homeless session... 2024-03-29T11:15:33.658 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.680+0000 7f0c0de4d640 10 client.4340.objecter successfully canceled tick 2024-03-29T11:15:33.658 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.680+0000 7f0c0de4d640 1 -- 172.21.15.47:0/3795102741 >> v1:172.21.15.121:6816/1964117370 conn(0x7f0bf406dcc0 legacy=0x7f0bf4070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:33.658 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.680+0000 7f0c0de4d640 10 monclient: shutdown 2024-03-29T11:15:33.658 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.680+0000 7f0c0de4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:33.658 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.680+0000 7f0c0de4d640 1 -- 172.21.15.47:0/3795102741 >> v1:172.21.15.47:6790/0 conn(0x7f0c08143d10 legacy=0x7f0c0812ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:33.658 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.680+0000 7f0c0de4d640 1 -- 172.21.15.47:0/3795102741 shutdown_connections 2024-03-29T11:15:33.658 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.680+0000 7f0c0de4d640 1 -- 172.21.15.47:0/3795102741 >> 172.21.15.47:0/3795102741 conn(0x7f0c08066a00 msgr2=0x7f0c080b7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:33.658 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.681+0000 7f0c0de4d640 1 -- 172.21.15.47:0/3795102741 shutdown_connections 2024-03-29T11:15:33.658 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.681+0000 7f0c0de4d640 1 -- 172.21.15.47:0/3795102741 wait complete. 2024-03-29T11:15:33.659 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:33.682+0000 7f0c0de4d640 1 librados: shutdown 2024-03-29T11:15:33.676 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:15:40.483 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:15:40.640 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.663+0000 7f6bdf740640 10 monclient: get_monmap_and_config 2024-03-29T11:15:40.640 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.663+0000 7f6bdf740640 10 monclient: build_initial_monmap 2024-03-29T11:15:40.641 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.663+0000 7f6bdf740640 10 monclient: monmap: 2024-03-29T11:15:40.641 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:15:40.641 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:15:40.641 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:15:40.664213+0000 2024-03-29T11:15:40.641 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:15:40.664213+0000 2024-03-29T11:15:40.641 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:40.641 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:15:40.641 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:15:40.641 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:15:40.641 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:15:40.641 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:15:40.644 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.665+0000 7f6bdf740640 1 Processor -- start 2024-03-29T11:15:40.644 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.666+0000 7f6bdf740640 1 -- start start 2024-03-29T11:15:40.644 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.666+0000 7f6bdf740640 10 monclient: init 2024-03-29T11:15:40.644 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.666+0000 7f6bdf740640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:40.644 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.666+0000 7f6bdf740640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:15:40.644 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.666+0000 7f6bdf740640 10 monclient(hunting): picked mon.noname-b con 0x7f6bd80b1f60 addr v1:172.21.15.47:6790/0 2024-03-29T11:15:40.644 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.666+0000 7f6bdf740640 10 monclient(hunting): picked mon.noname-a con 0x7f6bd8071eb0 addr v1:172.21.15.47:6789/0 2024-03-29T11:15:40.648 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.670+0000 7f6bdf740640 10 monclient(hunting): picked mon.noname-c con 0x7f6bd8143d10 addr v1:172.21.15.121:6789/0 2024-03-29T11:15:40.648 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.670+0000 7f6bdf740640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bd806ef80 con 0x7f6bd8071eb0 2024-03-29T11:15:40.648 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.670+0000 7f6bdf740640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bd806f260 con 0x7f6bd8143d10 2024-03-29T11:15:40.648 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.670+0000 7f6bdf740640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bd80b40c0 con 0x7f6bd80b1f60 2024-03-29T11:15:40.648 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.670+0000 7f6bdf740640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:40.648 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.670+0000 7f6bdf740640 10 monclient(hunting): authenticate will time out at 983.858752s 2024-03-29T11:15:40.648 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.671+0000 7f6bdd4dc640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f6bd80b1f60 0x7f6bd8143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:58635/0 (socket says 172.21.15.47:58635) 2024-03-29T11:15:40.649 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.671+0000 7f6bdd4dc640 1 -- 172.21.15.47:0/2470481156 learned_addr learned my addr 172.21.15.47:0/2470481156 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:15:40.650 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.672+0000 7f6bcffff640 1 -- 172.21.15.47:0/2470481156 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1738379444 0 0) 0x7f6bd80b40c0 con 0x7f6bd80b1f60 2024-03-29T11:15:40.650 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.672+0000 7f6bcffff640 10 monclient(hunting): _negotiate 2024-03-29T11:15:40.650 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.672+0000 7f6bcffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:40.650 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.672+0000 7f6bcffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:40.650 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.672+0000 7f6bcffff640 10 monclient(hunting): my global_id is 4343 2024-03-29T11:15:40.650 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.672+0000 7f6bcffff640 1 -- 172.21.15.47:0/2470481156 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bc4003520 con 0x7f6bd80b1f60 2024-03-29T11:15:40.650 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.672+0000 7f6bcffff640 1 -- 172.21.15.47:0/2470481156 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1153187238 0 0) 0x7f6bc4003520 con 0x7f6bd80b1f60 2024-03-29T11:15:40.651 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.672+0000 7f6bcffff640 1 -- 172.21.15.47:0/2470481156 >> v1:172.21.15.121:6789/0 conn(0x7f6bd8143d10 legacy=0x7f6bd8146b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:15:40.654 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.674+0000 7f6bcffff640 1 -- 172.21.15.47:0/2470481156 >> v1:172.21.15.47:6789/0 conn(0x7f6bd8071eb0 legacy=0x7f6bd8146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:40.654 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.674+0000 7f6bcffff640 10 monclient: _finish_hunting 0 2024-03-29T11:15:40.654 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.674+0000 7f6bcffff640 1 monclient: found mon.noname-b 2024-03-29T11:15:40.654 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.674+0000 7f6bcffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:40.654 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.674+0000 7f6bcffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-29T11:15:40.654 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.675+0000 7f6bcffff640 1 -- 172.21.15.47:0/2470481156 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bd8155d70 con 0x7f6bd80b1f60 2024-03-29T11:15:40.654 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.675+0000 7f6bcffff640 10 monclient: _finish_auth 0 2024-03-29T11:15:40.654 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.675+0000 7f6bcffff640 10 monclient: _check_auth_tickets 2024-03-29T11:15:40.654 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.675+0000 7f6bcffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:40.654 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.675+0000 7f6bcffff640 1 -- 172.21.15.47:0/2470481156 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f6bd4001690 con 0x7f6bd80b1f60 2024-03-29T11:15:40.655 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.675+0000 7f6bdf740640 5 monclient: authenticate success, global_id 4343 2024-03-29T11:15:40.655 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.675+0000 7f6bdf740640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:15:40.655 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.675+0000 7f6bcffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:40.655 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.675+0000 7f6bcffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:15:40.655 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.675+0000 7f6bcffff640 10 monclient: dump: 2024-03-29T11:15:40.655 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:15:40.655 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:15:40.655 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:40.655 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:40.655 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:40.655 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:15:40.655 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:15:40.655 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:15:40.655 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:15:40.655 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:15:40.656 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.675+0000 7f6bdf740640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:15:40.656 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.675+0000 7f6bcffff640 1 -- 172.21.15.47:0/2470481156 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6bd4001860 con 0x7f6bd80b1f60 2024-03-29T11:15:40.656 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.675+0000 7f6bcffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:40.656 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.675+0000 7f6bcffff640 1 -- 172.21.15.47:0/2470481156 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f6bd4001cb0 con 0x7f6bd80b1f60 2024-03-29T11:15:40.656 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.675+0000 7f6bdf740640 10 monclient: get_monmap_and_config success 2024-03-29T11:15:40.656 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.675+0000 7f6bdf740640 4 set_mon_vals no callback set 2024-03-29T11:15:40.656 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.675+0000 7f6bcffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:40.656 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.675+0000 7f6bcffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:15:40.656 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.675+0000 7f6bcffff640 10 monclient: dump: 2024-03-29T11:15:40.656 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:15:40.657 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:15:40.657 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:40.657 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:40.657 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:40.657 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:15:40.657 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:15:40.657 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:15:40.657 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:15:40.657 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:15:40.657 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.676+0000 7f6bdf740640 10 monclient: shutdown 2024-03-29T11:15:40.657 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.676+0000 7f6bdf740640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:40.657 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.676+0000 7f6bdf740640 1 -- 172.21.15.47:0/2470481156 >> v1:172.21.15.47:6790/0 conn(0x7f6bd80b1f60 legacy=0x7f6bd8143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:40.658 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.680+0000 7f6bdf740640 1 -- 172.21.15.47:0/2470481156 shutdown_connections 2024-03-29T11:15:40.658 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.680+0000 7f6bdf740640 1 -- 172.21.15.47:0/2470481156 >> 172.21.15.47:0/2470481156 conn(0x7f6bd8066a00 msgr2=0x7f6bd8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:40.658 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.680+0000 7f6bdf740640 1 -- 172.21.15.47:0/2470481156 shutdown_connections 2024-03-29T11:15:40.659 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.681+0000 7f6bdf740640 1 -- 172.21.15.47:0/2470481156 wait complete. 2024-03-29T11:15:40.661 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.682+0000 7f6bdf740640 10 monclient: build_initial_monmap 2024-03-29T11:15:40.661 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.682+0000 7f6bdf740640 10 monclient: monmap: 2024-03-29T11:15:40.661 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:15:40.661 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:15:40.661 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:15:40.662 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:15:40.662 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:40.662 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:15:40.662 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:15:40.662 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:15:40.662 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:15:40.662 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:15:40.662 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.682+0000 7f6bdf740640 1 librados: starting msgr at 2024-03-29T11:15:40.662 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.682+0000 7f6bdf740640 1 librados: starting objecter 2024-03-29T11:15:40.662 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.683+0000 7f6bdf740640 1 Processor -- start 2024-03-29T11:15:40.662 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.683+0000 7f6bdf740640 1 -- start start 2024-03-29T11:15:40.662 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.683+0000 7f6bdf740640 1 librados: setting wanted keys 2024-03-29T11:15:40.662 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.683+0000 7f6bdf740640 1 librados: calling monclient init 2024-03-29T11:15:40.663 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.683+0000 7f6bdf740640 10 monclient: init 2024-03-29T11:15:40.663 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.684+0000 7f6bdf740640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:40.663 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.684+0000 7f6bdf740640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:15:40.663 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.684+0000 7f6bdf740640 10 monclient(hunting): picked mon.noname-b con 0x7f6bd80b1f60 addr v1:172.21.15.121:6789/0 2024-03-29T11:15:40.663 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.686+0000 7f6bdf740640 10 monclient(hunting): picked mon.noname-c con 0x7f6bd8143d10 addr v1:172.21.15.47:6790/0 2024-03-29T11:15:40.669 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.689+0000 7f6bdd4dc640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f6bd80b1f60 0x7f6bd81dedc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.47:49435/0 (socket says 172.21.15.47:49435) 2024-03-29T11:15:40.669 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.690+0000 7f6bddcdd640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f6bd8154a80 0x7f6bd81e6300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:45371/0 (socket says 172.21.15.47:45371) 2024-03-29T11:15:40.669 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.690+0000 7f6bdf740640 10 monclient(hunting): picked mon.noname-a con 0x7f6bd8154a80 addr v1:172.21.15.47:6789/0 2024-03-29T11:15:40.669 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.690+0000 7f6bdd4dc640 1 -- 172.21.15.47:0/1121111696 learned_addr learned my addr 172.21.15.47:0/1121111696 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:15:40.669 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.690+0000 7f6bdf740640 1 -- 172.21.15.47:0/1121111696 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bd814b300 con 0x7f6bd8154a80 2024-03-29T11:15:40.670 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.690+0000 7f6bdf740640 1 -- 172.21.15.47:0/1121111696 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bd814b5e0 con 0x7f6bd80b1f60 2024-03-29T11:15:40.670 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.690+0000 7f6bdf740640 1 -- 172.21.15.47:0/1121111696 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bd81403f0 con 0x7f6bd8143d10 2024-03-29T11:15:40.670 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.690+0000 7f6bdf740640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:40.670 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.690+0000 7f6bdf740640 10 monclient(hunting): authenticate will time out at 983.878297s 2024-03-29T11:15:40.670 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.690+0000 7f6bcdffb640 10 client.?.objecter ms_handle_connect 0x7f6bd8154a80 2024-03-29T11:15:40.670 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.690+0000 7f6bcdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:40.670 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.690+0000 7f6bcdffb640 1 -- 172.21.15.47:0/1121111696 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3788126201 0 0) 0x7f6bd814b300 con 0x7f6bd8154a80 2024-03-29T11:15:40.670 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.690+0000 7f6bcdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:15:40.670 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.690+0000 7f6bcdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:40.670 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.690+0000 7f6bcdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:40.670 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.690+0000 7f6bcdffb640 10 monclient(hunting): my global_id is 4323 2024-03-29T11:15:40.670 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.690+0000 7f6bcdffb640 1 -- 172.21.15.47:0/1121111696 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bbc004eb0 con 0x7f6bd8154a80 2024-03-29T11:15:40.671 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.691+0000 7f6bcdffb640 1 -- 172.21.15.47:0/1121111696 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2921297375 0 0) 0x7f6bbc004eb0 con 0x7f6bd8154a80 2024-03-29T11:15:40.671 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.691+0000 7f6bcdffb640 1 -- 172.21.15.47:0/1121111696 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6bd814b300 con 0x7f6bd8154a80 2024-03-29T11:15:40.671 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.691+0000 7f6bcdffb640 1 -- 172.21.15.47:0/1121111696 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f6bd0001ad0 con 0x7f6bd8154a80 2024-03-29T11:15:40.671 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.691+0000 7f6bcdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:40.671 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.691+0000 7f6bcdffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:15:40.671 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.691+0000 7f6bcdffb640 10 monclient(hunting): dump: 2024-03-29T11:15:40.671 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:15:40.671 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:15:40.671 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:40.671 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:40.671 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:40.671 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:15:40.671 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:15:40.671 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:15:40.672 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:15:40.672 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:15:40.672 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.691+0000 7f6bcdffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:15:40.672 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.692+0000 7f6bcdffb640 10 client.?.objecter ms_handle_connect 0x7f6bd80b1f60 2024-03-29T11:15:40.672 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.692+0000 7f6bcdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:40.674 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.692+0000 7f6bcdffb640 1 -- 172.21.15.47:0/1121111696 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1294303943 0 0) 0x7f6bd814b5e0 con 0x7f6bd80b1f60 2024-03-29T11:15:40.674 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.692+0000 7f6bcdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:15:40.674 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.692+0000 7f6bcdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:40.674 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.692+0000 7f6bcdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:40.674 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.692+0000 7f6bcdffb640 10 monclient(hunting): my global_id is 4330 2024-03-29T11:15:40.674 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.692+0000 7f6bcdffb640 1 -- 172.21.15.47:0/1121111696 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bbc004eb0 con 0x7f6bd80b1f60 2024-03-29T11:15:40.675 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.693+0000 7f6bcdffb640 1 -- 172.21.15.47:0/1121111696 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3287985368 0 0) 0x7f6bbc004eb0 con 0x7f6bd80b1f60 2024-03-29T11:15:40.675 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.693+0000 7f6bcdffb640 1 -- 172.21.15.47:0/1121111696 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6bbc0059e0 con 0x7f6bd80b1f60 2024-03-29T11:15:40.675 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.693+0000 7f6bcdffb640 1 -- 172.21.15.47:0/1121111696 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f6bd4001b70 con 0x7f6bd80b1f60 2024-03-29T11:15:40.675 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.693+0000 7f6bcdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:40.675 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.693+0000 7f6bcdffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:15:40.675 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.693+0000 7f6bcdffb640 10 monclient(hunting): dump: 2024-03-29T11:15:40.675 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:15:40.675 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:15:40.675 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:40.675 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:40.675 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:40.675 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:15:40.675 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:15:40.675 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:15:40.676 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:15:40.676 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:15:40.676 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.693+0000 7f6bcdffb640 1 -- 172.21.15.47:0/1121111696 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3835364082 0 0) 0x7f6bd814b300 con 0x7f6bd8154a80 2024-03-29T11:15:40.676 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.693+0000 7f6bcdffb640 1 -- 172.21.15.47:0/1121111696 >> v1:172.21.15.47:6790/0 conn(0x7f6bd8143d10 legacy=0x7f6bd81df4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:40.676 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.693+0000 7f6bcdffb640 1 -- 172.21.15.47:0/1121111696 >> v1:172.21.15.121:6789/0 conn(0x7f6bd80b1f60 legacy=0x7f6bd81dedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:40.676 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.693+0000 7f6bcdffb640 10 monclient: _finish_hunting 0 2024-03-29T11:15:40.676 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.693+0000 7f6bcdffb640 1 monclient: found mon.a 2024-03-29T11:15:40.676 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.693+0000 7f6bcdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:40.676 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.693+0000 7f6bcdffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:15:40.676 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.693+0000 7f6bcdffb640 1 -- 172.21.15.47:0/1121111696 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bd813fd10 con 0x7f6bd8154a80 2024-03-29T11:15:40.676 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.693+0000 7f6bcdffb640 10 monclient: _finish_auth 0 2024-03-29T11:15:40.676 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.693+0000 7f6bcdffb640 10 monclient: _check_auth_tickets 2024-03-29T11:15:40.676 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.693+0000 7f6bcdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:40.676 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.694+0000 7f6bdf740640 5 monclient: authenticate success, global_id 4323 2024-03-29T11:15:40.677 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.694+0000 7f6bdf740640 10 monclient: _renew_subs 2024-03-29T11:15:40.677 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.694+0000 7f6bdf740640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:15:40.677 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.694+0000 7f6bdf740640 1 -- 172.21.15.47:0/1121111696 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bd813ff60 con 0x7f6bd8154a80 2024-03-29T11:15:40.677 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.694+0000 7f6bdf740640 10 client.4323.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:15:40.677 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.694+0000 7f6bdf740640 10 monclient: _renew_subs 2024-03-29T11:15:40.677 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.694+0000 7f6bdf740640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:15:40.677 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.694+0000 7f6bdf740640 1 -- 172.21.15.47:0/1121111696 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bd81e7c00 con 0x7f6bd8154a80 2024-03-29T11:15:40.677 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.694+0000 7f6bdf740640 1 librados: init done 2024-03-29T11:15:40.677 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.694+0000 7f6bcdffb640 1 -- 172.21.15.47:0/1121111696 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6bd00020a0 con 0x7f6bd8154a80 2024-03-29T11:15:40.677 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.694+0000 7f6bcdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:40.677 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.694+0000 7f6bcdffb640 1 -- 172.21.15.47:0/1121111696 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f6bd0004cc0 con 0x7f6bd8154a80 2024-03-29T11:15:40.677 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.694+0000 7f6bcdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:40.677 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.694+0000 7f6bcdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:15:40.677 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.694+0000 7f6bcdffb640 10 monclient: dump: 2024-03-29T11:15:40.677 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:15:40.677 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:15:40.678 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:40.678 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:40.678 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:40.678 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:15:40.678 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:15:40.678 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:15:40.678 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:15:40.678 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:15:40.678 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.695+0000 7f6bceffd640 4 set_mon_vals no callback set 2024-03-29T11:15:40.678 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.696+0000 7f6bdf740640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:40.678 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.696+0000 7f6bdf740640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:40.678 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.696+0000 7f6bdf740640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:15:40.678 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.696+0000 7f6bdf740640 1 -- 172.21.15.47:0/1121111696 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bb8005080 con 0x7f6bd8154a80 2024-03-29T11:15:40.679 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.698+0000 7f6bcdffb640 1 -- 172.21.15.47:0/1121111696 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7f6bd0004e20 con 0x7f6bd8154a80 2024-03-29T11:15:40.680 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.699+0000 7f6bcdffb640 1 -- 172.21.15.47:0/1121111696 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7f6bd0084ce0 con 0x7f6bd8154a80 2024-03-29T11:15:40.680 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.699+0000 7f6bcdffb640 10 client.4323.objecter ms_dispatch 0x7f6bd8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:15:40.680 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.699+0000 7f6bcdffb640 3 client.4323.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:15:40.680 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.699+0000 7f6bcdffb640 3 client.4323.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:15:40.680 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.699+0000 7f6bcdffb640 20 client.4323.objecter dump_active .. 0 homeless 2024-03-29T11:15:40.680 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.700+0000 7f6bdd4dc640 10 monclient: get_auth_request con 0x7f6bbc06e020 auth_method 0 2024-03-29T11:15:40.680 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.700+0000 7f6bcdffb640 10 client.4323.objecter ms_handle_connect 0x7f6bbc06e020 2024-03-29T11:15:40.684 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.707+0000 7f6bcdffb640 1 -- 172.21.15.47:0/1121111696 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6bd00503f0 con 0x7f6bd8154a80 2024-03-29T11:15:40.684 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.707+0000 7f6bcdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:40.684 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:40.707+0000 7f6bcdffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:15:41.024 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:41.046+0000 7f6bdf740640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:41.024 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:41.046+0000 7f6bdf740640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:41.024 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:41.046+0000 7f6bdf740640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:15:41.024 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:41.046+0000 7f6bdf740640 1 -- 172.21.15.47:0/1121111696 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6bb8005330 con 0x7f6bd8154a80 2024-03-29T11:15:41.024 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:41.047+0000 7f6bcdffb640 1 -- 172.21.15.47:0/1121111696 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7f6bd00524f0 con 0x7f6bd8154a80 2024-03-29T11:15:41.025 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:41.047+0000 7f6bcdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:41.025 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:41.047+0000 7f6bcdffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:15:41.025 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:15:41.026 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:15:41.029 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:41.051+0000 7f6bdf740640 10 librados: watch_flush enter 2024-03-29T11:15:41.029 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:41.051+0000 7f6bdf740640 10 librados: watch_flush exit 2024-03-29T11:15:41.029 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:41.051+0000 7f6bdf740640 20 client.4323.objecter shutdown clearing up homeless session... 2024-03-29T11:15:41.029 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:41.051+0000 7f6bdf740640 10 client.4323.objecter successfully canceled tick 2024-03-29T11:15:41.029 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:41.051+0000 7f6bdf740640 1 -- 172.21.15.47:0/1121111696 >> v1:172.21.15.121:6816/1964117370 conn(0x7f6bbc06e020 legacy=0x7f6bbc0704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:41.029 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:41.052+0000 7f6bdf740640 10 monclient: shutdown 2024-03-29T11:15:41.029 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:41.052+0000 7f6bdf740640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:41.030 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:41.052+0000 7f6bdf740640 1 -- 172.21.15.47:0/1121111696 >> v1:172.21.15.47:6789/0 conn(0x7f6bd8154a80 legacy=0x7f6bd81e6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:41.030 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:41.052+0000 7f6bdf740640 1 -- 172.21.15.47:0/1121111696 shutdown_connections 2024-03-29T11:15:41.030 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:41.052+0000 7f6bdf740640 1 -- 172.21.15.47:0/1121111696 >> 172.21.15.47:0/1121111696 conn(0x7f6bd8066a00 msgr2=0x7f6bd8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:41.032 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:41.055+0000 7f6bdf740640 1 -- 172.21.15.47:0/1121111696 shutdown_connections 2024-03-29T11:15:41.032 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:41.055+0000 7f6bdf740640 1 -- 172.21.15.47:0/1121111696 wait complete. 2024-03-29T11:15:41.032 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:41.055+0000 7f6bdf740640 1 librados: shutdown 2024-03-29T11:15:41.049 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:15:47.856 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:15:48.017 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.039+0000 7fc9fdc86640 10 monclient: get_monmap_and_config 2024-03-29T11:15:48.017 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.039+0000 7fc9fdc86640 10 monclient: build_initial_monmap 2024-03-29T11:15:48.017 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.039+0000 7fc9fdc86640 10 monclient: monmap: 2024-03-29T11:15:48.017 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:15:48.017 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:15:48.017 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:15:48.040532+0000 2024-03-29T11:15:48.017 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:15:48.040532+0000 2024-03-29T11:15:48.017 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:48.017 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:15:48.018 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:15:48.018 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:15:48.018 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:15:48.018 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:15:48.020 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.043+0000 7fc9fdc86640 1 Processor -- start 2024-03-29T11:15:48.021 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.043+0000 7fc9fdc86640 1 -- start start 2024-03-29T11:15:48.021 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.043+0000 7fc9fdc86640 10 monclient: init 2024-03-29T11:15:48.023 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.044+0000 7fc9fdc86640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:48.023 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.044+0000 7fc9fdc86640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:15:48.023 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.044+0000 7fc9fdc86640 10 monclient(hunting): picked mon.noname-b con 0x7fc9f806d9d0 addr v1:172.21.15.47:6790/0 2024-03-29T11:15:48.023 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.044+0000 7fc9fdc86640 10 monclient(hunting): picked mon.noname-c con 0x7fc9f813e820 addr v1:172.21.15.121:6789/0 2024-03-29T11:15:48.023 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.044+0000 7fc9fdc86640 10 monclient(hunting): picked mon.noname-a con 0x7fc9f80af1d0 addr v1:172.21.15.47:6789/0 2024-03-29T11:15:48.023 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.044+0000 7fc9fdc86640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9f80aed10 con 0x7fc9f80af1d0 2024-03-29T11:15:48.023 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.044+0000 7fc9fdc86640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9f80afd50 con 0x7fc9f813e820 2024-03-29T11:15:48.023 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.044+0000 7fc9fdc86640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9f80ab490 con 0x7fc9f806d9d0 2024-03-29T11:15:48.024 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.044+0000 7fc9fdc86640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:48.024 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.044+0000 7fc9fdc86640 10 monclient(hunting): authenticate will time out at 991.232405s 2024-03-29T11:15:48.027 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.050+0000 7fc9f7fff640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fc9f80af1d0 0x7fc9f80af640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:45829/0 (socket says 172.21.15.47:45829) 2024-03-29T11:15:48.028 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.050+0000 7fc9f7fff640 1 -- 172.21.15.47:0/3209197958 learned_addr learned my addr 172.21.15.47:0/3209197958 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:15:48.028 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.051+0000 7fc9f67fc640 1 -- 172.21.15.47:0/3209197958 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1284387183 0 0) 0x7fc9f80aed10 con 0x7fc9f80af1d0 2024-03-29T11:15:48.028 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.051+0000 7fc9f67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:15:48.028 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.051+0000 7fc9f67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:48.028 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.051+0000 7fc9f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:48.028 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.051+0000 7fc9f67fc640 10 monclient(hunting): my global_id is 4326 2024-03-29T11:15:48.029 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.051+0000 7fc9f67fc640 1 -- 172.21.15.47:0/3209197958 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9e0003520 con 0x7fc9f80af1d0 2024-03-29T11:15:48.029 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.051+0000 7fc9f67fc640 1 -- 172.21.15.47:0/3209197958 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 159166015 0 0) 0x7fc9e0003520 con 0x7fc9f80af1d0 2024-03-29T11:15:48.030 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.051+0000 7fc9f67fc640 1 -- 172.21.15.47:0/3209197958 >> v1:172.21.15.47:6790/0 conn(0x7fc9f806d9d0 legacy=0x7fc9f80b05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:15:48.030 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.051+0000 7fc9f67fc640 1 -- 172.21.15.47:0/3209197958 >> v1:172.21.15.121:6789/0 conn(0x7fc9f813e820 legacy=0x7fc9f813ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:48.030 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.051+0000 7fc9f67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:15:48.030 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.051+0000 7fc9f67fc640 1 monclient: found mon.noname-a 2024-03-29T11:15:48.030 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.051+0000 7fc9f67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:48.030 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.051+0000 7fc9f67fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-29T11:15:48.030 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.051+0000 7fc9f67fc640 1 -- 172.21.15.47:0/3209197958 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9f80ab770 con 0x7fc9f80af1d0 2024-03-29T11:15:48.030 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.051+0000 7fc9f67fc640 10 monclient: _finish_auth 0 2024-03-29T11:15:48.030 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.051+0000 7fc9f67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:15:48.031 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.051+0000 7fc9f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:48.031 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.051+0000 7fc9f67fc640 1 -- 172.21.15.47:0/3209197958 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fc9f0001a10 con 0x7fc9f80af1d0 2024-03-29T11:15:48.031 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.051+0000 7fc9f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:48.031 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.051+0000 7fc9f67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:15:48.031 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.051+0000 7fc9f67fc640 10 monclient: dump: 2024-03-29T11:15:48.031 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:15:48.031 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:15:48.031 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:48.031 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:48.031 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:48.031 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:15:48.031 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:15:48.031 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:15:48.031 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:15:48.031 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:15:48.032 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.051+0000 7fc9fdc86640 5 monclient: authenticate success, global_id 4326 2024-03-29T11:15:48.032 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.051+0000 7fc9fdc86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:15:48.032 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.051+0000 7fc9f67fc640 1 -- 172.21.15.47:0/3209197958 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc9f0001be0 con 0x7fc9f80af1d0 2024-03-29T11:15:48.032 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.051+0000 7fc9f67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:48.032 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.052+0000 7fc9fdc86640 10 monclient: get_monmap_and_config success 2024-03-29T11:15:48.032 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.052+0000 7fc9fdc86640 4 set_mon_vals no callback set 2024-03-29T11:15:48.032 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.052+0000 7fc9f67fc640 1 -- 172.21.15.47:0/3209197958 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fc9f0001ff0 con 0x7fc9f80af1d0 2024-03-29T11:15:48.032 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.052+0000 7fc9f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:48.032 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.052+0000 7fc9f67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:15:48.032 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.052+0000 7fc9f67fc640 10 monclient: dump: 2024-03-29T11:15:48.032 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:15:48.032 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:15:48.032 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:48.032 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:48.033 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:48.033 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:15:48.033 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:15:48.033 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:15:48.033 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:15:48.033 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:15:48.033 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.053+0000 7fc9fdc86640 10 monclient: shutdown 2024-03-29T11:15:48.033 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.053+0000 7fc9fdc86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:48.033 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.053+0000 7fc9fdc86640 1 -- 172.21.15.47:0/3209197958 >> v1:172.21.15.47:6789/0 conn(0x7fc9f80af1d0 legacy=0x7fc9f80af640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:48.033 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.053+0000 7fc9fdc86640 1 -- 172.21.15.47:0/3209197958 shutdown_connections 2024-03-29T11:15:48.033 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.053+0000 7fc9fdc86640 1 -- 172.21.15.47:0/3209197958 >> 172.21.15.47:0/3209197958 conn(0x7fc9f806b660 msgr2=0x7fc9f806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:48.034 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.056+0000 7fc9fdc86640 1 -- 172.21.15.47:0/3209197958 shutdown_connections 2024-03-29T11:15:48.034 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.056+0000 7fc9fdc86640 1 -- 172.21.15.47:0/3209197958 wait complete. 2024-03-29T11:15:48.034 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.057+0000 7fc9fdc86640 10 monclient: build_initial_monmap 2024-03-29T11:15:48.034 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.057+0000 7fc9fdc86640 10 monclient: monmap: 2024-03-29T11:15:48.035 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:15:48.035 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:15:48.035 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:15:48.035 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:15:48.035 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:48.035 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:15:48.035 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:15:48.035 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:15:48.035 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:15:48.035 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:15:48.035 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.057+0000 7fc9fdc86640 1 librados: starting msgr at 2024-03-29T11:15:48.035 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.057+0000 7fc9fdc86640 1 librados: starting objecter 2024-03-29T11:15:48.035 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.057+0000 7fc9fdc86640 1 Processor -- start 2024-03-29T11:15:48.037 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.058+0000 7fc9fdc86640 1 -- start start 2024-03-29T11:15:48.039 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.058+0000 7fc9fdc86640 1 librados: setting wanted keys 2024-03-29T11:15:48.039 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.058+0000 7fc9fdc86640 1 librados: calling monclient init 2024-03-29T11:15:48.039 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.058+0000 7fc9fdc86640 10 monclient: init 2024-03-29T11:15:48.039 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.059+0000 7fc9fdc86640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:48.039 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.059+0000 7fc9fdc86640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:15:48.039 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.059+0000 7fc9fdc86640 10 monclient(hunting): picked mon.noname-a con 0x7fc9f806d9d0 addr v1:172.21.15.47:6789/0 2024-03-29T11:15:48.039 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.059+0000 7fc9fdc86640 10 monclient(hunting): picked mon.noname-c con 0x7fc9f80af1d0 addr v1:172.21.15.47:6790/0 2024-03-29T11:15:48.039 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.059+0000 7fc9fdc86640 10 monclient(hunting): picked mon.noname-b con 0x7fc9f813e820 addr v1:172.21.15.121:6789/0 2024-03-29T11:15:48.039 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.059+0000 7fc9fdc86640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9f8070db0 con 0x7fc9f806d9d0 2024-03-29T11:15:48.039 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.059+0000 7fc9fdc86640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9f806fa50 con 0x7fc9f813e820 2024-03-29T11:15:48.040 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.059+0000 7fc9fdc86640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9f806fcc0 con 0x7fc9f80af1d0 2024-03-29T11:15:48.040 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.059+0000 7fc9fdc86640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:48.040 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.059+0000 7fc9fdc86640 10 monclient(hunting): authenticate will time out at 991.247450s 2024-03-29T11:15:48.044 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.066+0000 7fc9f6ffd640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fc9f80af1d0 0x7fc9f81ebff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:39525/0 (socket says 172.21.15.47:39525) 2024-03-29T11:15:48.044 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.066+0000 7fc9f6ffd640 1 -- 172.21.15.47:0/420544400 learned_addr learned my addr 172.21.15.47:0/420544400 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:15:48.044 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.066+0000 7fc9dffff640 10 client.?.objecter ms_handle_connect 0x7fc9f80af1d0 2024-03-29T11:15:48.044 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.066+0000 7fc9dffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:48.044 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.066+0000 7fc9dffff640 1 -- 172.21.15.47:0/420544400 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 395209423 0 0) 0x7fc9f806fcc0 con 0x7fc9f80af1d0 2024-03-29T11:15:48.044 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.066+0000 7fc9dffff640 10 monclient(hunting): _negotiate 2024-03-29T11:15:48.044 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.066+0000 7fc9dffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:48.044 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.066+0000 7fc9dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:48.045 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.066+0000 7fc9dffff640 10 monclient(hunting): my global_id is 4346 2024-03-29T11:15:48.045 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.066+0000 7fc9dffff640 1 -- 172.21.15.47:0/420544400 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9e4004eb0 con 0x7fc9f80af1d0 2024-03-29T11:15:48.051 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.070+0000 7fc9dffff640 10 client.?.objecter ms_handle_connect 0x7fc9f806d9d0 2024-03-29T11:15:48.051 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.070+0000 7fc9dffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:48.051 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.070+0000 7fc9dffff640 1 -- 172.21.15.47:0/420544400 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2055695225 0 0) 0x7fc9e4004eb0 con 0x7fc9f80af1d0 2024-03-29T11:15:48.051 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.070+0000 7fc9dffff640 1 -- 172.21.15.47:0/420544400 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc9f806fcc0 con 0x7fc9f80af1d0 2024-03-29T11:15:48.051 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.070+0000 7fc9dffff640 1 -- 172.21.15.47:0/420544400 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1044271237 0 0) 0x7fc9f8070db0 con 0x7fc9f806d9d0 2024-03-29T11:15:48.051 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.070+0000 7fc9dffff640 10 monclient(hunting): _negotiate 2024-03-29T11:15:48.051 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.070+0000 7fc9dffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:48.051 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.070+0000 7fc9dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:48.051 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.070+0000 7fc9dffff640 10 monclient(hunting): my global_id is 4329 2024-03-29T11:15:48.051 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.070+0000 7fc9dffff640 1 -- 172.21.15.47:0/420544400 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9e4004eb0 con 0x7fc9f806d9d0 2024-03-29T11:15:48.052 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.070+0000 7fc9dffff640 1 -- 172.21.15.47:0/420544400 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fc9ec0019d0 con 0x7fc9f80af1d0 2024-03-29T11:15:48.052 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.070+0000 7fc9dffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:48.052 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.070+0000 7fc9dffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:15:48.052 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.070+0000 7fc9dffff640 10 monclient(hunting): dump: 2024-03-29T11:15:48.052 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:15:48.052 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:15:48.052 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:48.052 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:48.052 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:48.052 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:15:48.052 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:15:48.052 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:15:48.052 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:15:48.052 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:15:48.053 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.070+0000 7fc9dffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:15:48.053 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.071+0000 7fc9dffff640 1 -- 172.21.15.47:0/420544400 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 258435550 0 0) 0x7fc9e4004eb0 con 0x7fc9f806d9d0 2024-03-29T11:15:48.053 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.071+0000 7fc9dffff640 1 -- 172.21.15.47:0/420544400 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc9f8070db0 con 0x7fc9f806d9d0 2024-03-29T11:15:48.053 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.071+0000 7fc9dffff640 1 -- 172.21.15.47:0/420544400 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fc9e8001690 con 0x7fc9f806d9d0 2024-03-29T11:15:48.053 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.071+0000 7fc9dffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:48.053 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.071+0000 7fc9dffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:15:48.053 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.071+0000 7fc9dffff640 10 monclient(hunting): dump: 2024-03-29T11:15:48.053 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:15:48.053 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:15:48.053 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:48.053 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:48.053 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:48.053 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:15:48.053 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:15:48.054 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:15:48.054 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:15:48.054 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:15:48.054 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.071+0000 7fc9dffff640 1 -- 172.21.15.47:0/420544400 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 600186596 0 0) 0x7fc9f8070db0 con 0x7fc9f806d9d0 2024-03-29T11:15:48.054 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.071+0000 7fc9dffff640 1 -- 172.21.15.47:0/420544400 >> v1:172.21.15.47:6790/0 conn(0x7fc9f80af1d0 legacy=0x7fc9f81ebff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:48.054 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.071+0000 7fc9dffff640 1 -- 172.21.15.47:0/420544400 >> v1:172.21.15.121:6789/0 conn(0x7fc9f813e820 legacy=0x7fc9f81ef720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:48.054 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.071+0000 7fc9dffff640 10 monclient: _finish_hunting 0 2024-03-29T11:15:48.054 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.071+0000 7fc9dffff640 1 monclient: found mon.a 2024-03-29T11:15:48.054 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.071+0000 7fc9dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:48.054 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.071+0000 7fc9dffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:15:48.054 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.071+0000 7fc9dffff640 1 -- 172.21.15.47:0/420544400 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9f806ffa0 con 0x7fc9f806d9d0 2024-03-29T11:15:48.054 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.071+0000 7fc9dffff640 10 monclient: _finish_auth 0 2024-03-29T11:15:48.054 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.071+0000 7fc9dffff640 10 monclient: _check_auth_tickets 2024-03-29T11:15:48.054 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.071+0000 7fc9dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:48.055 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.071+0000 7fc9fdc86640 5 monclient: authenticate success, global_id 4329 2024-03-29T11:15:48.055 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.071+0000 7fc9fdc86640 10 monclient: _renew_subs 2024-03-29T11:15:48.055 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.071+0000 7fc9fdc86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:15:48.055 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.071+0000 7fc9fdc86640 1 -- 172.21.15.47:0/420544400 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9f8070280 con 0x7fc9f806d9d0 2024-03-29T11:15:48.055 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.071+0000 7fc9fdc86640 10 client.4329.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:15:48.055 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.071+0000 7fc9fdc86640 10 monclient: _renew_subs 2024-03-29T11:15:48.055 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.071+0000 7fc9fdc86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:15:48.055 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.071+0000 7fc9fdc86640 1 -- 172.21.15.47:0/420544400 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9f806e420 con 0x7fc9f806d9d0 2024-03-29T11:15:48.055 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.071+0000 7fc9fdc86640 1 librados: init done 2024-03-29T11:15:48.057 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.074+0000 7fc9dffff640 1 -- 172.21.15.47:0/420544400 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc9e8001ba0 con 0x7fc9f806d9d0 2024-03-29T11:15:48.057 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.074+0000 7fc9dffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:48.057 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.075+0000 7fc9f57fa640 4 set_mon_vals no callback set 2024-03-29T11:15:48.057 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.075+0000 7fc9fdc86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:48.057 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.075+0000 7fc9dffff640 1 -- 172.21.15.47:0/420544400 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fc9e8003790 con 0x7fc9f806d9d0 2024-03-29T11:15:48.057 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.075+0000 7fc9dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:48.057 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.075+0000 7fc9dffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:15:48.058 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.075+0000 7fc9dffff640 10 monclient: dump: 2024-03-29T11:15:48.058 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:15:48.058 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:15:48.058 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:48.058 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:48.058 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:48.058 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:15:48.058 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:15:48.058 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:15:48.058 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:15:48.058 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:15:48.058 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.075+0000 7fc9dffff640 1 -- 172.21.15.47:0/420544400 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7fc9e8019760 con 0x7fc9f806d9d0 2024-03-29T11:15:48.058 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.076+0000 7fc9dffff640 1 -- 172.21.15.47:0/420544400 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7fc9e8083740 con 0x7fc9f806d9d0 2024-03-29T11:15:48.058 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.076+0000 7fc9dffff640 10 client.4329.objecter ms_dispatch 0x7fc9f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:15:48.059 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.076+0000 7fc9dffff640 3 client.4329.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:15:48.059 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.076+0000 7fc9dffff640 3 client.4329.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:15:48.059 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.076+0000 7fc9dffff640 20 client.4329.objecter dump_active .. 0 homeless 2024-03-29T11:15:48.059 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.077+0000 7fc9fdc86640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:48.059 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.077+0000 7fc9fdc86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:15:48.059 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.077+0000 7fc9fdc86640 1 -- 172.21.15.47:0/420544400 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9d8005080 con 0x7fc9f806d9d0 2024-03-29T11:15:48.059 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.077+0000 7fc9f6ffd640 10 monclient: get_auth_request con 0x7fc9e406df60 auth_method 0 2024-03-29T11:15:48.059 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.078+0000 7fc9dffff640 10 client.4329.objecter ms_handle_connect 0x7fc9e406df60 2024-03-29T11:15:48.064 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.086+0000 7fc9dffff640 1 -- 172.21.15.47:0/420544400 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc9e804f0a0 con 0x7fc9f806d9d0 2024-03-29T11:15:48.064 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.086+0000 7fc9dffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:48.064 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.086+0000 7fc9dffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:15:48.392 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.414+0000 7fc9fdc86640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:48.392 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.414+0000 7fc9fdc86640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:48.392 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.414+0000 7fc9fdc86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:15:48.392 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.414+0000 7fc9fdc86640 1 -- 172.21.15.47:0/420544400 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc9d8005330 con 0x7fc9f806d9d0 2024-03-29T11:15:48.393 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.416+0000 7fc9dffff640 1 -- 172.21.15.47:0/420544400 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7fc9e8050fd0 con 0x7fc9f806d9d0 2024-03-29T11:15:48.393 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.416+0000 7fc9dffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:48.394 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.416+0000 7fc9dffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:15:48.394 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:15:48.394 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:15:48.396 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.419+0000 7fc9fdc86640 10 librados: watch_flush enter 2024-03-29T11:15:48.396 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.419+0000 7fc9fdc86640 10 librados: watch_flush exit 2024-03-29T11:15:48.396 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.419+0000 7fc9fdc86640 20 client.4329.objecter shutdown clearing up homeless session... 2024-03-29T11:15:48.396 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.419+0000 7fc9fdc86640 10 client.4329.objecter successfully canceled tick 2024-03-29T11:15:48.396 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.419+0000 7fc9fdc86640 1 -- 172.21.15.47:0/420544400 >> v1:172.21.15.121:6816/1964117370 conn(0x7fc9e406df60 legacy=0x7fc9e4070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:48.397 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.419+0000 7fc9fdc86640 10 monclient: shutdown 2024-03-29T11:15:48.397 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.419+0000 7fc9fdc86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:48.397 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.419+0000 7fc9fdc86640 1 -- 172.21.15.47:0/420544400 >> v1:172.21.15.47:6789/0 conn(0x7fc9f806d9d0 legacy=0x7fc9f80706a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:48.397 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.419+0000 7fc9fdc86640 1 -- 172.21.15.47:0/420544400 shutdown_connections 2024-03-29T11:15:48.397 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.419+0000 7fc9fdc86640 1 -- 172.21.15.47:0/420544400 >> 172.21.15.47:0/420544400 conn(0x7fc9f806b660 msgr2=0x7fc9f813ec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:48.397 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.420+0000 7fc9fdc86640 1 -- 172.21.15.47:0/420544400 shutdown_connections 2024-03-29T11:15:48.397 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.420+0000 7fc9fdc86640 1 -- 172.21.15.47:0/420544400 wait complete. 2024-03-29T11:15:48.398 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:48.421+0000 7fc9fdc86640 1 librados: shutdown 2024-03-29T11:15:48.416 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:15:55.224 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:15:55.377 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.399+0000 7fd7adcf1640 10 monclient: get_monmap_and_config 2024-03-29T11:15:55.377 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.399+0000 7fd7adcf1640 10 monclient: build_initial_monmap 2024-03-29T11:15:55.377 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.399+0000 7fd7adcf1640 10 monclient: monmap: 2024-03-29T11:15:55.378 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:15:55.378 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:15:55.378 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:15:55.400909+0000 2024-03-29T11:15:55.378 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:15:55.400909+0000 2024-03-29T11:15:55.378 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:55.378 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:15:55.378 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:15:55.378 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:15:55.378 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:15:55.378 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:15:55.379 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.401+0000 7fd7adcf1640 1 Processor -- start 2024-03-29T11:15:55.379 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.401+0000 7fd7adcf1640 1 -- start start 2024-03-29T11:15:55.379 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.401+0000 7fd7adcf1640 10 monclient: init 2024-03-29T11:15:55.380 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.402+0000 7fd7adcf1640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:55.381 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.402+0000 7fd7adcf1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:15:55.381 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.402+0000 7fd7adcf1640 10 monclient(hunting): picked mon.noname-a con 0x7fd7a81457f0 addr v1:172.21.15.47:6789/0 2024-03-29T11:15:55.381 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.402+0000 7fd7adcf1640 10 monclient(hunting): picked mon.noname-c con 0x7fd7a813f650 addr v1:172.21.15.121:6789/0 2024-03-29T11:15:55.381 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.402+0000 7fd7adcf1640 10 monclient(hunting): picked mon.noname-b con 0x7fd7a80a9d70 addr v1:172.21.15.47:6790/0 2024-03-29T11:15:55.381 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.402+0000 7fd7adcf1640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a80aaef0 con 0x7fd7a81457f0 2024-03-29T11:15:55.381 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.402+0000 7fd7adcf1640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a8145bd0 con 0x7fd7a813f650 2024-03-29T11:15:55.381 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.402+0000 7fd7adcf1640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a8145eb0 con 0x7fd7a80a9d70 2024-03-29T11:15:55.381 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.402+0000 7fd7adcf1640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:55.381 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.402+0000 7fd7adcf1640 10 monclient(hunting): authenticate will time out at 998.590630s 2024-03-29T11:15:55.381 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.403+0000 7fd7a77fe640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fd7a81457f0 0x7fd7a813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:42690/0 (socket says 172.21.15.47:42690) 2024-03-29T11:15:55.381 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.403+0000 7fd7a77fe640 1 -- 172.21.15.47:0/4093676608 learned_addr learned my addr 172.21.15.47:0/4093676608 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:15:55.381 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.403+0000 7fd7a6ffd640 1 -- 172.21.15.47:0/4093676608 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2819071133 0 0) 0x7fd7a80aaef0 con 0x7fd7a81457f0 2024-03-29T11:15:55.381 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.403+0000 7fd7a6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:15:55.382 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.403+0000 7fd7a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:55.382 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.403+0000 7fd7a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:55.382 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.403+0000 7fd7a6ffd640 10 monclient(hunting): my global_id is 4332 2024-03-29T11:15:55.382 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.403+0000 7fd7a6ffd640 1 -- 172.21.15.47:0/4093676608 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd794003520 con 0x7fd7a81457f0 2024-03-29T11:15:55.383 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.404+0000 7fd7a6ffd640 1 -- 172.21.15.47:0/4093676608 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2847361711 0 0) 0x7fd794003520 con 0x7fd7a81457f0 2024-03-29T11:15:55.383 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.404+0000 7fd7a6ffd640 1 -- 172.21.15.47:0/4093676608 >> v1:172.21.15.47:6790/0 conn(0x7fd7a80a9d70 legacy=0x7fd7a814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:55.383 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.404+0000 7fd7a6ffd640 1 -- 172.21.15.47:0/4093676608 >> v1:172.21.15.121:6789/0 conn(0x7fd7a813f650 legacy=0x7fd7a80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:55.383 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.404+0000 7fd7a6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:15:55.383 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.404+0000 7fd7a6ffd640 1 monclient: found mon.noname-a 2024-03-29T11:15:55.383 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.404+0000 7fd7a6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:55.383 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.404+0000 7fd7a6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-29T11:15:55.383 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.404+0000 7fd7a6ffd640 1 -- 172.21.15.47:0/4093676608 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7a8146340 con 0x7fd7a81457f0 2024-03-29T11:15:55.383 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.404+0000 7fd7a6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:15:55.383 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.404+0000 7fd7a6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:15:55.383 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.404+0000 7fd7a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:55.383 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.404+0000 7fd7a6ffd640 1 -- 172.21.15.47:0/4093676608 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fd798001690 con 0x7fd7a81457f0 2024-03-29T11:15:55.384 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.404+0000 7fd7a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:55.384 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.404+0000 7fd7a6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:15:55.384 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.404+0000 7fd7a6ffd640 10 monclient: dump: 2024-03-29T11:15:55.384 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:15:55.384 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:15:55.384 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:55.384 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:55.384 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:55.384 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:15:55.384 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:15:55.384 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:15:55.384 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:15:55.384 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:15:55.384 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.404+0000 7fd7adcf1640 5 monclient: authenticate success, global_id 4332 2024-03-29T11:15:55.385 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.404+0000 7fd7adcf1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:15:55.385 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.404+0000 7fd7a6ffd640 1 -- 172.21.15.47:0/4093676608 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd7980017f0 con 0x7fd7a81457f0 2024-03-29T11:15:55.385 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.404+0000 7fd7a6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:55.385 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.404+0000 7fd7a6ffd640 1 -- 172.21.15.47:0/4093676608 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fd798001c40 con 0x7fd7a81457f0 2024-03-29T11:15:55.385 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.404+0000 7fd7a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:55.385 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.404+0000 7fd7a6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:15:55.385 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.404+0000 7fd7a6ffd640 10 monclient: dump: 2024-03-29T11:15:55.385 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:15:55.385 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:15:55.385 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:55.385 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:55.385 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:55.385 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:15:55.385 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:15:55.385 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:15:55.386 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:15:55.386 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:15:55.386 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.404+0000 7fd7adcf1640 10 monclient: get_monmap_and_config success 2024-03-29T11:15:55.386 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.404+0000 7fd7adcf1640 4 set_mon_vals no callback set 2024-03-29T11:15:55.386 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.405+0000 7fd7adcf1640 10 monclient: shutdown 2024-03-29T11:15:55.386 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.405+0000 7fd7adcf1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:55.386 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.405+0000 7fd7adcf1640 1 -- 172.21.15.47:0/4093676608 >> v1:172.21.15.47:6789/0 conn(0x7fd7a81457f0 legacy=0x7fd7a813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:55.386 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.405+0000 7fd7adcf1640 1 -- 172.21.15.47:0/4093676608 shutdown_connections 2024-03-29T11:15:55.386 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.405+0000 7fd7adcf1640 1 -- 172.21.15.47:0/4093676608 >> 172.21.15.47:0/4093676608 conn(0x7fd7a8066a00 msgr2=0x7fd7a8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:55.386 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.405+0000 7fd7adcf1640 1 -- 172.21.15.47:0/4093676608 shutdown_connections 2024-03-29T11:15:55.386 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.405+0000 7fd7adcf1640 1 -- 172.21.15.47:0/4093676608 wait complete. 2024-03-29T11:15:55.389 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.406+0000 7fd7adcf1640 10 monclient: build_initial_monmap 2024-03-29T11:15:55.390 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.406+0000 7fd7adcf1640 10 monclient: monmap: 2024-03-29T11:15:55.390 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:15:55.390 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:15:55.390 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:15:55.390 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:15:55.390 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:55.390 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:15:55.390 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:15:55.390 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:15:55.390 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:15:55.390 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:15:55.390 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.407+0000 7fd7adcf1640 1 librados: starting msgr at 2024-03-29T11:15:55.390 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.407+0000 7fd7adcf1640 1 librados: starting objecter 2024-03-29T11:15:55.391 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.407+0000 7fd7adcf1640 1 Processor -- start 2024-03-29T11:15:55.391 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.407+0000 7fd7adcf1640 1 -- start start 2024-03-29T11:15:55.391 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.407+0000 7fd7adcf1640 1 librados: setting wanted keys 2024-03-29T11:15:55.391 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.407+0000 7fd7adcf1640 1 librados: calling monclient init 2024-03-29T11:15:55.391 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.407+0000 7fd7adcf1640 10 monclient: init 2024-03-29T11:15:55.391 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.408+0000 7fd7adcf1640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:55.391 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.408+0000 7fd7adcf1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:15:55.391 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.408+0000 7fd7adcf1640 10 monclient(hunting): picked mon.noname-c con 0x7fd7a80a9d70 addr v1:172.21.15.47:6790/0 2024-03-29T11:15:55.391 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.408+0000 7fd7adcf1640 10 monclient(hunting): picked mon.noname-a con 0x7fd7a813f650 addr v1:172.21.15.47:6789/0 2024-03-29T11:15:55.391 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.408+0000 7fd7a77fe640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fd7a80a9d70 0x7fd7a81446c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:47858/0 (socket says 172.21.15.47:47858) 2024-03-29T11:15:55.391 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.408+0000 7fd7adcf1640 10 monclient(hunting): picked mon.noname-b con 0x7fd7a81457f0 addr v1:172.21.15.121:6789/0 2024-03-29T11:15:55.391 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.408+0000 7fd7a77fe640 1 -- 172.21.15.47:0/3804559833 learned_addr learned my addr 172.21.15.47:0/3804559833 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:15:55.391 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.408+0000 7fd7adcf1640 1 -- 172.21.15.47:0/3804559833 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a806e780 con 0x7fd7a813f650 2024-03-29T11:15:55.391 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.408+0000 7fd7adcf1640 1 -- 172.21.15.47:0/3804559833 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a806efb0 con 0x7fd7a81457f0 2024-03-29T11:15:55.392 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.408+0000 7fd7adcf1640 1 -- 172.21.15.47:0/3804559833 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a8071180 con 0x7fd7a80a9d70 2024-03-29T11:15:55.392 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.408+0000 7fd7adcf1640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:55.392 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.408+0000 7fd7adcf1640 10 monclient(hunting): authenticate will time out at 998.596766s 2024-03-29T11:15:55.392 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.409+0000 7fd7a4ff9640 10 client.?.objecter ms_handle_connect 0x7fd7a80a9d70 2024-03-29T11:15:55.392 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.409+0000 7fd7a4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:55.392 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.409+0000 7fd7a4ff9640 1 -- 172.21.15.47:0/3804559833 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3315111329 0 0) 0x7fd7a8071180 con 0x7fd7a80a9d70 2024-03-29T11:15:55.392 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.409+0000 7fd7a4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:15:55.392 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.409+0000 7fd7a4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:55.392 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.409+0000 7fd7a4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:55.392 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.409+0000 7fd7a4ff9640 10 monclient(hunting): my global_id is 4352 2024-03-29T11:15:55.392 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.409+0000 7fd7a4ff9640 1 -- 172.21.15.47:0/3804559833 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd788004eb0 con 0x7fd7a80a9d70 2024-03-29T11:15:55.392 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.409+0000 7fd7a4ff9640 10 client.?.objecter ms_handle_connect 0x7fd7a813f650 2024-03-29T11:15:55.392 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.409+0000 7fd7a4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:55.392 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.410+0000 7fd7a4ff9640 10 client.?.objecter ms_handle_connect 0x7fd7a81457f0 2024-03-29T11:15:55.393 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.410+0000 7fd7a4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:55.393 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.410+0000 7fd7a4ff9640 1 -- 172.21.15.47:0/3804559833 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3356743943 0 0) 0x7fd7a806e780 con 0x7fd7a813f650 2024-03-29T11:15:55.393 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.410+0000 7fd7a4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:15:55.393 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.410+0000 7fd7a4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:55.393 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.410+0000 7fd7a4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:55.393 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.410+0000 7fd7a4ff9640 10 monclient(hunting): my global_id is 4335 2024-03-29T11:15:55.393 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.410+0000 7fd7a4ff9640 1 -- 172.21.15.47:0/3804559833 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7a8071180 con 0x7fd7a813f650 2024-03-29T11:15:55.393 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.410+0000 7fd7a4ff9640 1 -- 172.21.15.47:0/3804559833 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1813244560 0 0) 0x7fd788004eb0 con 0x7fd7a80a9d70 2024-03-29T11:15:55.393 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.410+0000 7fd7a4ff9640 1 -- 172.21.15.47:0/3804559833 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd7a806e780 con 0x7fd7a80a9d70 2024-03-29T11:15:55.393 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.410+0000 7fd7a4ff9640 1 -- 172.21.15.47:0/3804559833 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fd798001c00 con 0x7fd7a80a9d70 2024-03-29T11:15:55.393 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.410+0000 7fd7a4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:55.393 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.410+0000 7fd7a4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:15:55.393 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.410+0000 7fd7a4ff9640 10 monclient(hunting): dump: 2024-03-29T11:15:55.393 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:15:55.394 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:15:55.394 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:55.394 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:55.394 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:55.394 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:15:55.394 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:15:55.394 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:15:55.394 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:15:55.394 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:15:55.394 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.410+0000 7fd7a4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:15:55.394 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.410+0000 7fd7a4ff9640 1 -- 172.21.15.47:0/3804559833 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 242470732 0 0) 0x7fd7a806efb0 con 0x7fd7a81457f0 2024-03-29T11:15:55.394 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.410+0000 7fd7a4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:15:55.394 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.410+0000 7fd7a4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:55.394 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.410+0000 7fd7a4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:55.395 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.410+0000 7fd7a4ff9640 10 monclient(hunting): my global_id is 4339 2024-03-29T11:15:55.395 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.410+0000 7fd7a4ff9640 1 -- 172.21.15.47:0/3804559833 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd788004eb0 con 0x7fd7a81457f0 2024-03-29T11:15:55.395 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.411+0000 7fd7a4ff9640 1 -- 172.21.15.47:0/3804559833 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 778561363 0 0) 0x7fd7a8071180 con 0x7fd7a813f650 2024-03-29T11:15:55.395 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.411+0000 7fd7a4ff9640 1 -- 172.21.15.47:0/3804559833 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd7a806efb0 con 0x7fd7a813f650 2024-03-29T11:15:55.395 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.411+0000 7fd7a4ff9640 1 -- 172.21.15.47:0/3804559833 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fd7900018a0 con 0x7fd7a813f650 2024-03-29T11:15:55.395 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.411+0000 7fd7a4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:55.395 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.411+0000 7fd7a4ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:15:55.395 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.411+0000 7fd7a4ff9640 10 monclient(hunting): dump: 2024-03-29T11:15:55.395 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:15:55.395 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:15:55.395 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:55.395 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:55.395 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:55.396 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:15:55.396 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:15:55.396 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:15:55.396 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:15:55.396 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:15:55.396 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.411+0000 7fd7a4ff9640 1 -- 172.21.15.47:0/3804559833 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1035920207 0 0) 0x7fd788004eb0 con 0x7fd7a81457f0 2024-03-29T11:15:55.396 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.411+0000 7fd7a4ff9640 1 -- 172.21.15.47:0/3804559833 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd7a8071180 con 0x7fd7a81457f0 2024-03-29T11:15:55.396 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.411+0000 7fd7a4ff9640 1 -- 172.21.15.47:0/3804559833 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fd7a0001bf0 con 0x7fd7a81457f0 2024-03-29T11:15:55.396 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.411+0000 7fd7a4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:55.396 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.411+0000 7fd7a4ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:15:55.396 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.411+0000 7fd7a4ff9640 10 monclient(hunting): dump: 2024-03-29T11:15:55.396 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:15:55.396 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:15:55.396 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:55.397 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:55.397 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:55.397 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:15:55.397 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:15:55.397 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:15:55.397 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:15:55.397 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:15:55.397 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.411+0000 7fd7a4ff9640 1 -- 172.21.15.47:0/3804559833 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 328311124 0 0) 0x7fd7a806e780 con 0x7fd7a80a9d70 2024-03-29T11:15:55.397 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.411+0000 7fd7a4ff9640 1 -- 172.21.15.47:0/3804559833 >> v1:172.21.15.121:6789/0 conn(0x7fd7a81457f0 legacy=0x7fd7a81e64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:55.397 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.411+0000 7fd7a4ff9640 1 -- 172.21.15.47:0/3804559833 >> v1:172.21.15.47:6789/0 conn(0x7fd7a813f650 legacy=0x7fd7a81424b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:55.397 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.411+0000 7fd7a4ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:15:55.397 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.411+0000 7fd7a4ff9640 1 monclient: found mon.c 2024-03-29T11:15:55.397 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.411+0000 7fd7a4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:55.397 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.411+0000 7fd7a4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:15:55.398 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.411+0000 7fd7a4ff9640 1 -- 172.21.15.47:0/3804559833 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7a806e9f0 con 0x7fd7a80a9d70 2024-03-29T11:15:55.398 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.411+0000 7fd7a4ff9640 10 monclient: _finish_auth 0 2024-03-29T11:15:55.398 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.411+0000 7fd7a4ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:15:55.398 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.411+0000 7fd7a4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:55.398 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.411+0000 7fd7adcf1640 5 monclient: authenticate success, global_id 4352 2024-03-29T11:15:55.398 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.411+0000 7fd7adcf1640 10 monclient: _renew_subs 2024-03-29T11:15:55.398 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.411+0000 7fd7adcf1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:15:55.398 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.411+0000 7fd7adcf1640 1 -- 172.21.15.47:0/3804559833 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7a806ecd0 con 0x7fd7a80a9d70 2024-03-29T11:15:55.398 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.411+0000 7fd7adcf1640 10 client.4352.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:15:55.398 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.411+0000 7fd7adcf1640 10 monclient: _renew_subs 2024-03-29T11:15:55.398 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.411+0000 7fd7adcf1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:15:55.398 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.411+0000 7fd7adcf1640 1 -- 172.21.15.47:0/3804559833 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7a8071ca0 con 0x7fd7a80a9d70 2024-03-29T11:15:55.398 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.411+0000 7fd7adcf1640 1 librados: init done 2024-03-29T11:15:55.399 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.412+0000 7fd7a4ff9640 1 -- 172.21.15.47:0/3804559833 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd798004a90 con 0x7fd7a80a9d70 2024-03-29T11:15:55.399 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.412+0000 7fd7a4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:55.399 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.412+0000 7fd7a4ff9640 1 -- 172.21.15.47:0/3804559833 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fd798004da0 con 0x7fd7a80a9d70 2024-03-29T11:15:55.399 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.412+0000 7fd7a4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:55.399 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.412+0000 7fd7a4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:15:55.399 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.412+0000 7fd7a5ffb640 4 set_mon_vals no callback set 2024-03-29T11:15:55.399 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.412+0000 7fd7a4ff9640 10 monclient: dump: 2024-03-29T11:15:55.399 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:15:55.399 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:15:55.399 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:55.399 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:15:55.399 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:55.399 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:15:55.399 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:15:55.399 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:15:55.400 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:15:55.400 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:15:55.401 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.413+0000 7fd7adcf1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:55.401 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.413+0000 7fd7adcf1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:55.401 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.413+0000 7fd7adcf1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:15:55.401 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.413+0000 7fd7adcf1640 1 -- 172.21.15.47:0/3804559833 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd78c005080 con 0x7fd7a80a9d70 2024-03-29T11:15:55.401 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.413+0000 7fd7a4ff9640 1 -- 172.21.15.47:0/3804559833 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7fd79801ad70 con 0x7fd7a80a9d70 2024-03-29T11:15:55.401 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.414+0000 7fd7a4ff9640 1 -- 172.21.15.47:0/3804559833 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7fd798084ba0 con 0x7fd7a80a9d70 2024-03-29T11:15:55.401 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.414+0000 7fd7a4ff9640 10 client.4352.objecter ms_dispatch 0x7fd7a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:15:55.401 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.414+0000 7fd7a4ff9640 3 client.4352.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:15:55.401 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.414+0000 7fd7a4ff9640 3 client.4352.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:15:55.401 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.414+0000 7fd7a4ff9640 20 client.4352.objecter dump_active .. 0 homeless 2024-03-29T11:15:55.401 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.414+0000 7fd79effd640 10 monclient: get_auth_request con 0x7fd78806e120 auth_method 0 2024-03-29T11:15:55.401 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.414+0000 7fd7a4ff9640 10 client.4352.objecter ms_handle_connect 0x7fd78806e120 2024-03-29T11:15:55.402 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.424+0000 7fd7a4ff9640 1 -- 172.21.15.47:0/3804559833 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd798050480 con 0x7fd7a80a9d70 2024-03-29T11:15:55.402 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.424+0000 7fd7a4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:55.402 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.424+0000 7fd7a4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:15:55.780 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.801+0000 7fd7adcf1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:55.780 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.801+0000 7fd7adcf1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:55.780 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.801+0000 7fd7adcf1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:15:55.780 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.801+0000 7fd7adcf1640 1 -- 172.21.15.47:0/3804559833 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd78c005330 con 0x7fd7a80a9d70 2024-03-29T11:15:55.781 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.803+0000 7fd7a4ff9640 1 -- 172.21.15.47:0/3804559833 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7fd7980523b0 con 0x7fd7a80a9d70 2024-03-29T11:15:55.781 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.803+0000 7fd7a4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:55.781 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.803+0000 7fd7a4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:15:55.781 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:15:55.781 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:15:55.784 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.807+0000 7fd7adcf1640 10 librados: watch_flush enter 2024-03-29T11:15:55.784 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.807+0000 7fd7adcf1640 10 librados: watch_flush exit 2024-03-29T11:15:55.784 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.807+0000 7fd7adcf1640 20 client.4352.objecter shutdown clearing up homeless session... 2024-03-29T11:15:55.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.807+0000 7fd7adcf1640 10 client.4352.objecter successfully canceled tick 2024-03-29T11:15:55.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.807+0000 7fd7adcf1640 1 -- 172.21.15.47:0/3804559833 >> v1:172.21.15.121:6816/1964117370 conn(0x7fd78806e120 legacy=0x7fd7880705e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:55.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.807+0000 7fd7adcf1640 10 monclient: shutdown 2024-03-29T11:15:55.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.807+0000 7fd7adcf1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:55.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.807+0000 7fd7adcf1640 1 -- 172.21.15.47:0/3804559833 >> v1:172.21.15.47:6790/0 conn(0x7fd7a80a9d70 legacy=0x7fd7a81446c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:55.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.807+0000 7fd7adcf1640 1 -- 172.21.15.47:0/3804559833 shutdown_connections 2024-03-29T11:15:55.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.807+0000 7fd7adcf1640 1 -- 172.21.15.47:0/3804559833 >> 172.21.15.47:0/3804559833 conn(0x7fd7a8066a00 msgr2=0x7fd7a8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:55.787 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.810+0000 7fd7adcf1640 1 -- 172.21.15.47:0/3804559833 shutdown_connections 2024-03-29T11:15:55.788 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.811+0000 7fd7adcf1640 1 -- 172.21.15.47:0/3804559833 wait complete. 2024-03-29T11:15:55.788 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:15:55.811+0000 7fd7adcf1640 1 librados: shutdown 2024-03-29T11:15:55.810 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:16:02.616 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:16:02.770 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.792+0000 7ff78ce57640 10 monclient: get_monmap_and_config 2024-03-29T11:16:02.771 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.792+0000 7ff78ce57640 10 monclient: build_initial_monmap 2024-03-29T11:16:02.771 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.792+0000 7ff78ce57640 10 monclient: monmap: 2024-03-29T11:16:02.771 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:16:02.771 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:16:02.771 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:16:02.793685+0000 2024-03-29T11:16:02.771 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:16:02.793685+0000 2024-03-29T11:16:02.771 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:02.771 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:16:02.771 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:16:02.771 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:16:02.771 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:16:02.771 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:16:02.771 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.793+0000 7ff78ce57640 1 Processor -- start 2024-03-29T11:16:02.771 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.793+0000 7ff78ce57640 1 -- start start 2024-03-29T11:16:02.771 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.793+0000 7ff78ce57640 10 monclient: init 2024-03-29T11:16:02.775 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.794+0000 7ff78ce57640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:02.775 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.794+0000 7ff78ce57640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:16:02.775 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.794+0000 7ff78ce57640 10 monclient(hunting): picked mon.noname-c con 0x7ff7880b0500 addr v1:172.21.15.121:6789/0 2024-03-29T11:16:02.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.794+0000 7ff78ce57640 10 monclient(hunting): picked mon.noname-b con 0x7ff7880b08e0 addr v1:172.21.15.47:6790/0 2024-03-29T11:16:02.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.794+0000 7ff78ce57640 10 monclient(hunting): picked mon.noname-a con 0x7ff7880aeca0 addr v1:172.21.15.47:6789/0 2024-03-29T11:16:02.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.794+0000 7ff78ce57640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff788140620 con 0x7ff7880aeca0 2024-03-29T11:16:02.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.794+0000 7ff78ce57640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff78813daf0 con 0x7ff7880b0500 2024-03-29T11:16:02.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.794+0000 7ff78ce57640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7880ae180 con 0x7ff7880b08e0 2024-03-29T11:16:02.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.794+0000 7ff78ce57640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:02.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.794+0000 7ff78ce57640 10 monclient(hunting): authenticate will time out at 1005.982864s 2024-03-29T11:16:02.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.795+0000 7ff78659c640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7ff7880b0500 0x7ff78813ff10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.47:39010/0 (socket says 172.21.15.47:39010) 2024-03-29T11:16:02.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.795+0000 7ff78659c640 1 -- 172.21.15.47:0/3809472929 learned_addr learned my addr 172.21.15.47:0/3809472929 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:16:02.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.795+0000 7ff78559a640 1 -- 172.21.15.47:0/3809472929 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3032613420 0 0) 0x7ff78813daf0 con 0x7ff7880b0500 2024-03-29T11:16:02.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.795+0000 7ff78559a640 10 monclient(hunting): _negotiate 2024-03-29T11:16:02.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.795+0000 7ff78559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:02.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.795+0000 7ff78559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:02.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.795+0000 7ff78559a640 10 monclient(hunting): my global_id is 4342 2024-03-29T11:16:02.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.795+0000 7ff78559a640 1 -- 172.21.15.47:0/3809472929 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff770003520 con 0x7ff7880b0500 2024-03-29T11:16:02.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.795+0000 7ff78559a640 1 -- 172.21.15.47:0/3809472929 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2550370053 0 0) 0x7ff788140620 con 0x7ff7880aeca0 2024-03-29T11:16:02.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.795+0000 7ff78559a640 10 monclient(hunting): _negotiate 2024-03-29T11:16:02.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.795+0000 7ff78559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:02.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.795+0000 7ff78559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:02.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.795+0000 7ff78559a640 10 monclient(hunting): my global_id is 4338 2024-03-29T11:16:02.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.795+0000 7ff78559a640 1 -- 172.21.15.47:0/3809472929 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff78813daf0 con 0x7ff7880aeca0 2024-03-29T11:16:02.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.796+0000 7ff78559a640 1 -- 172.21.15.47:0/3809472929 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3904239302 0 0) 0x7ff7880ae180 con 0x7ff7880b08e0 2024-03-29T11:16:02.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.796+0000 7ff78559a640 10 monclient(hunting): _negotiate 2024-03-29T11:16:02.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.796+0000 7ff78559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:02.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.796+0000 7ff78559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:02.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.796+0000 7ff78559a640 10 monclient(hunting): my global_id is 4355 2024-03-29T11:16:02.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.796+0000 7ff78559a640 1 -- 172.21.15.47:0/3809472929 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff788140620 con 0x7ff7880b08e0 2024-03-29T11:16:02.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.796+0000 7ff78559a640 1 -- 172.21.15.47:0/3809472929 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2217561893 0 0) 0x7ff770003520 con 0x7ff7880b0500 2024-03-29T11:16:02.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.796+0000 7ff78559a640 1 -- 172.21.15.47:0/3809472929 >> v1:172.21.15.47:6790/0 conn(0x7ff7880b08e0 legacy=0x7ff78813d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:02.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.796+0000 7ff78559a640 1 -- 172.21.15.47:0/3809472929 >> v1:172.21.15.47:6789/0 conn(0x7ff7880aeca0 legacy=0x7ff78814b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:02.778 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.796+0000 7ff78559a640 10 monclient: _finish_hunting 0 2024-03-29T11:16:02.778 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.796+0000 7ff78559a640 1 monclient: found mon.noname-c 2024-03-29T11:16:02.778 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.796+0000 7ff78559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:02.778 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.796+0000 7ff78559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:16:02.778 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.796+0000 7ff78559a640 1 -- 172.21.15.47:0/3809472929 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7880accf0 con 0x7ff7880b0500 2024-03-29T11:16:02.778 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.796+0000 7ff78559a640 10 monclient: _finish_auth 0 2024-03-29T11:16:02.778 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.796+0000 7ff78559a640 10 monclient: _check_auth_tickets 2024-03-29T11:16:02.778 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.796+0000 7ff78559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:02.778 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.796+0000 7ff78559a640 1 -- 172.21.15.47:0/3809472929 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7ff778001690 con 0x7ff7880b0500 2024-03-29T11:16:02.778 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.796+0000 7ff78559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:02.778 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.796+0000 7ff78559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:16:02.778 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.796+0000 7ff78559a640 10 monclient: dump: 2024-03-29T11:16:02.778 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:16:02.778 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:16:02.779 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:02.779 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:02.779 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:02.779 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:16:02.779 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:16:02.779 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:16:02.779 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:16:02.779 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:16:02.779 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.796+0000 7ff78ce57640 5 monclient: authenticate success, global_id 4342 2024-03-29T11:16:02.779 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.796+0000 7ff78ce57640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:16:02.779 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.796+0000 7ff78559a640 1 -- 172.21.15.47:0/3809472929 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff778001870 con 0x7ff7880b0500 2024-03-29T11:16:02.779 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.796+0000 7ff78559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:02.779 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.796+0000 7ff78559a640 1 -- 172.21.15.47:0/3809472929 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7ff778001690 con 0x7ff7880b0500 2024-03-29T11:16:02.779 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.796+0000 7ff78559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:02.780 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.796+0000 7ff78559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:16:02.780 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.796+0000 7ff78559a640 10 monclient: dump: 2024-03-29T11:16:02.780 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:16:02.780 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:16:02.780 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:02.780 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:02.780 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:02.780 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:16:02.780 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:16:02.780 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:16:02.780 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:16:02.780 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:16:02.780 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.797+0000 7ff78ce57640 10 monclient: get_monmap_and_config success 2024-03-29T11:16:02.780 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.797+0000 7ff78ce57640 4 set_mon_vals no callback set 2024-03-29T11:16:02.780 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.797+0000 7ff78ce57640 10 monclient: shutdown 2024-03-29T11:16:02.781 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.797+0000 7ff78ce57640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:02.781 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.797+0000 7ff78ce57640 1 -- 172.21.15.47:0/3809472929 >> v1:172.21.15.121:6789/0 conn(0x7ff7880b0500 legacy=0x7ff78813ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:02.781 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.798+0000 7ff78ce57640 1 -- 172.21.15.47:0/3809472929 shutdown_connections 2024-03-29T11:16:02.781 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.798+0000 7ff78ce57640 1 -- 172.21.15.47:0/3809472929 >> 172.21.15.47:0/3809472929 conn(0x7ff78806b660 msgr2=0x7ff78806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:02.781 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.798+0000 7ff78ce57640 1 -- 172.21.15.47:0/3809472929 shutdown_connections 2024-03-29T11:16:02.781 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.798+0000 7ff78ce57640 1 -- 172.21.15.47:0/3809472929 wait complete. 2024-03-29T11:16:02.784 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.799+0000 7ff78ce57640 10 monclient: build_initial_monmap 2024-03-29T11:16:02.784 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.799+0000 7ff78ce57640 10 monclient: monmap: 2024-03-29T11:16:02.784 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:16:02.784 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:16:02.785 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:16:02.785 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:16:02.785 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:02.785 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:16:02.785 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:16:02.785 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:16:02.785 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:16:02.785 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:16:02.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.799+0000 7ff78ce57640 1 librados: starting msgr at 2024-03-29T11:16:02.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.799+0000 7ff78ce57640 1 librados: starting objecter 2024-03-29T11:16:02.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.799+0000 7ff78ce57640 1 Processor -- start 2024-03-29T11:16:02.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.800+0000 7ff78ce57640 1 -- start start 2024-03-29T11:16:02.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.800+0000 7ff78ce57640 1 librados: setting wanted keys 2024-03-29T11:16:02.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.800+0000 7ff78ce57640 1 librados: calling monclient init 2024-03-29T11:16:02.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.800+0000 7ff78ce57640 10 monclient: init 2024-03-29T11:16:02.786 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.800+0000 7ff78ce57640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:02.786 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.800+0000 7ff78ce57640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:16:02.786 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.800+0000 7ff78ce57640 10 monclient(hunting): picked mon.noname-b con 0x7ff7880aeca0 addr v1:172.21.15.121:6789/0 2024-03-29T11:16:02.786 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.801+0000 7ff78ce57640 10 monclient(hunting): picked mon.noname-a con 0x7ff7880b0500 addr v1:172.21.15.47:6789/0 2024-03-29T11:16:02.786 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.801+0000 7ff78ce57640 10 monclient(hunting): picked mon.noname-c con 0x7ff7881ceac0 addr v1:172.21.15.47:6790/0 2024-03-29T11:16:02.786 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.801+0000 7ff78ce57640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff78811cb10 con 0x7ff7880b0500 2024-03-29T11:16:02.786 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.801+0000 7ff785d9b640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7ff7880b0500 0x7ff7881de8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:42554/0 (socket says 172.21.15.47:42554) 2024-03-29T11:16:02.786 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.801+0000 7ff785d9b640 1 -- 172.21.15.47:0/3957894129 learned_addr learned my addr 172.21.15.47:0/3957894129 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:16:02.786 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.801+0000 7ff776ffd640 10 client.?.objecter ms_handle_connect 0x7ff7880b0500 2024-03-29T11:16:02.786 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.801+0000 7ff776ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:02.786 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.801+0000 7ff776ffd640 10 client.?.objecter ms_handle_connect 0x7ff7880aeca0 2024-03-29T11:16:02.786 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.801+0000 7ff776ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:02.786 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.801+0000 7ff78ce57640 1 -- 172.21.15.47:0/3957894129 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff78811def0 con 0x7ff7880aeca0 2024-03-29T11:16:02.786 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.801+0000 7ff78ce57640 1 -- 172.21.15.47:0/3957894129 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff78811bb70 con 0x7ff7881ceac0 2024-03-29T11:16:02.787 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.801+0000 7ff78ce57640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:02.787 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.801+0000 7ff78ce57640 10 monclient(hunting): authenticate will time out at 1005.989402s 2024-03-29T11:16:02.787 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.801+0000 7ff776ffd640 10 client.?.objecter ms_handle_connect 0x7ff7881ceac0 2024-03-29T11:16:02.787 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.801+0000 7ff776ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:02.787 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.802+0000 7ff776ffd640 1 -- 172.21.15.47:0/3957894129 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1871792318 0 0) 0x7ff78811def0 con 0x7ff7880aeca0 2024-03-29T11:16:02.787 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.802+0000 7ff776ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:16:02.787 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.802+0000 7ff776ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:02.787 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.802+0000 7ff776ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:02.787 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.802+0000 7ff776ffd640 10 monclient(hunting): my global_id is 4345 2024-03-29T11:16:02.787 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.802+0000 7ff776ffd640 1 -- 172.21.15.47:0/3957894129 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff768004eb0 con 0x7ff7880aeca0 2024-03-29T11:16:02.787 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.802+0000 7ff776ffd640 1 -- 172.21.15.47:0/3957894129 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 472268530 0 0) 0x7ff77c001210 con 0x7ff7880b0500 2024-03-29T11:16:02.787 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.802+0000 7ff776ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:16:02.787 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.802+0000 7ff776ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:02.787 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.802+0000 7ff776ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:02.787 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.802+0000 7ff776ffd640 10 monclient(hunting): my global_id is 4341 2024-03-29T11:16:02.788 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.802+0000 7ff776ffd640 1 -- 172.21.15.47:0/3957894129 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff78811def0 con 0x7ff7880b0500 2024-03-29T11:16:02.788 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.802+0000 7ff786d9d640 1 -- 172.21.15.47:0/3957894129 >> v1:172.21.15.47:6790/0 conn(0x7ff7881ceac0 legacy=0x7ff78813aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-29T11:16:02.788 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.802+0000 7ff786d9d640 1 -- 172.21.15.47:0/3957894129 >> v1:172.21.15.47:6790/0 conn(0x7ff7881ceac0 legacy=0x7ff78813aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:16:02.788 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.802+0000 7ff786d9d640 1 --1- 172.21.15.47:0/3957894129 >> v1:172.21.15.47:6790/0 conn(0x7ff7881ceac0 0x7ff78813aae0 :-1 s=OPENED pgs=321386 cs=1 l=1).handle_message read tag failed 2024-03-29T11:16:02.788 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.802+0000 7ff786d9d640 1 --1- 172.21.15.47:0/3957894129 >> v1:172.21.15.47:6790/0 conn(0x7ff7881ceac0 0x7ff78813aae0 :-1 s=OPENED pgs=321386 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:16:02.788 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.802+0000 7ff776ffd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.47:6790/0 2024-03-29T11:16:02.788 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.803+0000 7ff776ffd640 1 -- 172.21.15.47:0/3957894129 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3408612698 0 0) 0x7ff78811def0 con 0x7ff7880b0500 2024-03-29T11:16:02.788 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.803+0000 7ff776ffd640 1 -- 172.21.15.47:0/3957894129 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff77c001210 con 0x7ff7880b0500 2024-03-29T11:16:02.788 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.803+0000 7ff776ffd640 1 -- 172.21.15.47:0/3957894129 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7ff77c001900 con 0x7ff7880b0500 2024-03-29T11:16:02.788 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.803+0000 7ff776ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:02.788 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.803+0000 7ff776ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:16:02.788 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.803+0000 7ff776ffd640 10 monclient(hunting): dump: 2024-03-29T11:16:02.788 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:16:02.788 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:16:02.789 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:02.789 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:02.789 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:02.789 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:16:02.789 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:16:02.789 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:16:02.789 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:16:02.789 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:16:02.789 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.803+0000 7ff776ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:16:02.789 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.803+0000 7ff776ffd640 1 -- 172.21.15.47:0/3957894129 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2646142136 0 0) 0x7ff77c001210 con 0x7ff7880b0500 2024-03-29T11:16:02.789 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.803+0000 7ff776ffd640 1 -- 172.21.15.47:0/3957894129 >> v1:172.21.15.47:6790/0 conn(0x7ff7881ceac0 legacy=0x7ff78813aae0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:16:02.789 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.803+0000 7ff776ffd640 1 -- 172.21.15.47:0/3957894129 >> v1:172.21.15.121:6789/0 conn(0x7ff7880aeca0 legacy=0x7ff78811be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:02.789 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.803+0000 7ff776ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:16:02.789 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.803+0000 7ff776ffd640 1 monclient: found mon.a 2024-03-29T11:16:02.790 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.803+0000 7ff776ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:02.790 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.803+0000 7ff776ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:16:02.790 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.803+0000 7ff776ffd640 1 -- 172.21.15.47:0/3957894129 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff78811c5f0 con 0x7ff7880b0500 2024-03-29T11:16:02.790 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.803+0000 7ff776ffd640 10 monclient: _finish_auth 0 2024-03-29T11:16:02.790 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.803+0000 7ff776ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:16:02.790 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.803+0000 7ff776ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:02.790 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.803+0000 7ff78ce57640 5 monclient: authenticate success, global_id 4341 2024-03-29T11:16:02.790 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.803+0000 7ff78ce57640 10 monclient: _renew_subs 2024-03-29T11:16:02.790 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.803+0000 7ff78ce57640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:16:02.790 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.803+0000 7ff78ce57640 1 -- 172.21.15.47:0/3957894129 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff78813b1f0 con 0x7ff7880b0500 2024-03-29T11:16:02.790 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.804+0000 7ff78ce57640 10 client.4341.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:16:02.790 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.804+0000 7ff78ce57640 10 monclient: _renew_subs 2024-03-29T11:16:02.790 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.804+0000 7ff78ce57640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:16:02.790 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.804+0000 7ff78ce57640 1 -- 172.21.15.47:0/3957894129 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7881defe0 con 0x7ff7880b0500 2024-03-29T11:16:02.790 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.804+0000 7ff78ce57640 1 librados: init done 2024-03-29T11:16:02.791 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.805+0000 7ff776ffd640 1 -- 172.21.15.47:0/3957894129 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff77c003610 con 0x7ff7880b0500 2024-03-29T11:16:02.791 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.805+0000 7ff776ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:02.791 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.805+0000 7ff776ffd640 1 -- 172.21.15.47:0/3957894129 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7ff77c0039e0 con 0x7ff7880b0500 2024-03-29T11:16:02.791 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.805+0000 7ff776ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:02.791 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.805+0000 7ff776ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:16:02.791 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.805+0000 7ff776ffd640 10 monclient: dump: 2024-03-29T11:16:02.791 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:16:02.791 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:16:02.791 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:02.791 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:02.791 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:02.791 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:16:02.791 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:16:02.791 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:16:02.791 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:16:02.792 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:16:02.792 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.805+0000 7ff784d99640 4 set_mon_vals no callback set 2024-03-29T11:16:02.792 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.806+0000 7ff78ce57640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:02.792 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.806+0000 7ff78ce57640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:02.792 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.806+0000 7ff78ce57640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:16:02.792 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.806+0000 7ff78ce57640 1 -- 172.21.15.47:0/3957894129 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff76c005080 con 0x7ff7880b0500 2024-03-29T11:16:02.793 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.807+0000 7ff776ffd640 1 -- 172.21.15.47:0/3957894129 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7ff77c003b40 con 0x7ff7880b0500 2024-03-29T11:16:02.793 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.808+0000 7ff776ffd640 1 -- 172.21.15.47:0/3957894129 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7ff77c0848b0 con 0x7ff7880b0500 2024-03-29T11:16:02.793 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.808+0000 7ff776ffd640 10 client.4341.objecter ms_dispatch 0x7ff788000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:16:02.793 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.808+0000 7ff776ffd640 3 client.4341.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:16:02.793 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.808+0000 7ff776ffd640 3 client.4341.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:16:02.793 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.808+0000 7ff776ffd640 20 client.4341.objecter dump_active .. 0 homeless 2024-03-29T11:16:02.793 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.808+0000 7ff78659c640 10 monclient: get_auth_request con 0x7ff76806dcf0 auth_method 0 2024-03-29T11:16:02.793 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.808+0000 7ff776ffd640 10 client.4341.objecter ms_handle_connect 0x7ff76806dcf0 2024-03-29T11:16:02.794 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.817+0000 7ff776ffd640 1 -- 172.21.15.47:0/3957894129 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff77c0502d0 con 0x7ff7880b0500 2024-03-29T11:16:02.795 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.817+0000 7ff776ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:02.795 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:02.817+0000 7ff776ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:16:03.201 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:03.223+0000 7ff78ce57640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:03.201 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:03.223+0000 7ff78ce57640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:03.201 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:03.223+0000 7ff78ce57640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:16:03.201 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:03.223+0000 7ff78ce57640 1 -- 172.21.15.47:0/3957894129 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff76c005330 con 0x7ff7880b0500 2024-03-29T11:16:03.202 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:03.225+0000 7ff776ffd640 1 -- 172.21.15.47:0/3957894129 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7ff77c052200 con 0x7ff7880b0500 2024-03-29T11:16:03.203 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:03.225+0000 7ff776ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:03.203 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:03.225+0000 7ff776ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:16:03.203 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:16:03.203 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:16:03.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:03.229+0000 7ff78ce57640 10 librados: watch_flush enter 2024-03-29T11:16:03.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:03.229+0000 7ff78ce57640 10 librados: watch_flush exit 2024-03-29T11:16:03.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:03.229+0000 7ff78ce57640 20 client.4341.objecter shutdown clearing up homeless session... 2024-03-29T11:16:03.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:03.229+0000 7ff78ce57640 10 client.4341.objecter successfully canceled tick 2024-03-29T11:16:03.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:03.229+0000 7ff78ce57640 1 -- 172.21.15.47:0/3957894129 >> v1:172.21.15.121:6816/1964117370 conn(0x7ff76806dcf0 legacy=0x7ff7680701b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:03.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:03.229+0000 7ff78ce57640 10 monclient: shutdown 2024-03-29T11:16:03.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:03.229+0000 7ff78ce57640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:03.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:03.229+0000 7ff78ce57640 1 -- 172.21.15.47:0/3957894129 >> v1:172.21.15.47:6789/0 conn(0x7ff7880b0500 legacy=0x7ff7881de8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:03.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:03.229+0000 7ff78ce57640 1 -- 172.21.15.47:0/3957894129 shutdown_connections 2024-03-29T11:16:03.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:03.229+0000 7ff78ce57640 1 -- 172.21.15.47:0/3957894129 >> 172.21.15.47:0/3957894129 conn(0x7ff78806b660 msgr2=0x7ff7880b2cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:03.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:03.229+0000 7ff78ce57640 1 -- 172.21.15.47:0/3957894129 shutdown_connections 2024-03-29T11:16:03.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:03.229+0000 7ff78ce57640 1 -- 172.21.15.47:0/3957894129 wait complete. 2024-03-29T11:16:03.208 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:03.230+0000 7ff78ce57640 1 librados: shutdown 2024-03-29T11:16:03.222 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:16:10.030 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:16:10.185 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.207+0000 7f9876579640 10 monclient: get_monmap_and_config 2024-03-29T11:16:10.185 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.207+0000 7f9876579640 10 monclient: build_initial_monmap 2024-03-29T11:16:10.185 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.207+0000 7f9876579640 10 monclient: monmap: 2024-03-29T11:16:10.185 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:16:10.185 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:16:10.185 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:16:10.208102+0000 2024-03-29T11:16:10.185 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:16:10.208102+0000 2024-03-29T11:16:10.185 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:10.185 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:16:10.185 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:16:10.185 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:16:10.185 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:16:10.186 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:16:10.186 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.209+0000 7f9876579640 1 Processor -- start 2024-03-29T11:16:10.186 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.209+0000 7f9876579640 1 -- start start 2024-03-29T11:16:10.186 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.209+0000 7f9876579640 10 monclient: init 2024-03-29T11:16:10.188 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.210+0000 7f9876579640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:10.188 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.210+0000 7f9876579640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:16:10.189 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.210+0000 7f9876579640 10 monclient(hunting): picked mon.noname-c con 0x7f98700a9ac0 addr v1:172.21.15.121:6789/0 2024-03-29T11:16:10.189 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.210+0000 7f9876579640 10 monclient(hunting): picked mon.noname-a con 0x7f987013ee90 addr v1:172.21.15.47:6789/0 2024-03-29T11:16:10.189 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.210+0000 7f9876579640 10 monclient(hunting): picked mon.noname-b con 0x7f98700af900 addr v1:172.21.15.47:6790/0 2024-03-29T11:16:10.189 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.210+0000 7f9876579640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98700afd10 con 0x7f987013ee90 2024-03-29T11:16:10.189 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.210+0000 7f9876579640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98700af6f0 con 0x7f98700a9ac0 2024-03-29T11:16:10.189 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.210+0000 7f9876579640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98700aac80 con 0x7f98700af900 2024-03-29T11:16:10.189 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.210+0000 7f9876579640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:10.189 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.210+0000 7f9876579640 10 monclient(hunting): authenticate will time out at 1013.398673s 2024-03-29T11:16:10.189 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.211+0000 7f986f7fe640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f987013ee90 0x7f987013b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:58042/0 (socket says 172.21.15.47:58042) 2024-03-29T11:16:10.189 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.211+0000 7f986f7fe640 1 -- 172.21.15.47:0/3972229726 learned_addr learned my addr 172.21.15.47:0/3972229726 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:16:10.189 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.211+0000 7f986effd640 1 -- 172.21.15.47:0/3972229726 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 778816674 0 0) 0x7f98700aac80 con 0x7f98700af900 2024-03-29T11:16:10.189 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.211+0000 7f986effd640 10 monclient(hunting): _negotiate 2024-03-29T11:16:10.189 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.211+0000 7f986effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:10.189 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.211+0000 7f986effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:10.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.211+0000 7f986effd640 10 monclient(hunting): my global_id is 4361 2024-03-29T11:16:10.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.211+0000 7f986effd640 1 -- 172.21.15.47:0/3972229726 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9858003440 con 0x7f98700af900 2024-03-29T11:16:10.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.211+0000 7f986effd640 1 -- 172.21.15.47:0/3972229726 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1794077071 0 0) 0x7f98700afd10 con 0x7f987013ee90 2024-03-29T11:16:10.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.211+0000 7f986effd640 10 monclient(hunting): _negotiate 2024-03-29T11:16:10.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.211+0000 7f986effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:10.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.211+0000 7f986effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:10.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.211+0000 7f986effd640 10 monclient(hunting): my global_id is 4344 2024-03-29T11:16:10.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.211+0000 7f986effd640 1 -- 172.21.15.47:0/3972229726 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9858003930 con 0x7f987013ee90 2024-03-29T11:16:10.191 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.212+0000 7f986effd640 1 -- 172.21.15.47:0/3972229726 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3617516533 0 0) 0x7f9858003440 con 0x7f98700af900 2024-03-29T11:16:10.191 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.212+0000 7f986effd640 1 -- 172.21.15.47:0/3972229726 >> v1:172.21.15.121:6789/0 conn(0x7f98700a9ac0 legacy=0x7f98700adfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:16:10.191 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.213+0000 7f986effd640 1 -- 172.21.15.47:0/3972229726 >> v1:172.21.15.47:6789/0 conn(0x7f987013ee90 legacy=0x7f987013b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:10.191 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.213+0000 7f986effd640 10 monclient: _finish_hunting 0 2024-03-29T11:16:10.191 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.213+0000 7f986effd640 1 monclient: found mon.noname-b 2024-03-29T11:16:10.191 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.213+0000 7f986effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:10.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.213+0000 7f986effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-29T11:16:10.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.213+0000 7f986effd640 1 -- 172.21.15.47:0/3972229726 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98700aaf60 con 0x7f98700af900 2024-03-29T11:16:10.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.213+0000 7f986effd640 10 monclient: _finish_auth 0 2024-03-29T11:16:10.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.213+0000 7f986effd640 10 monclient: _check_auth_tickets 2024-03-29T11:16:10.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.213+0000 7f986effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:10.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.213+0000 7f986effd640 1 -- 172.21.15.47:0/3972229726 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f98680019f0 con 0x7f98700af900 2024-03-29T11:16:10.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.213+0000 7f986effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:10.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.213+0000 7f986effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:16:10.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.213+0000 7f986effd640 10 monclient: dump: 2024-03-29T11:16:10.192 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:16:10.192 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:16:10.192 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:10.192 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:10.192 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:10.192 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:16:10.193 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:16:10.193 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:16:10.193 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:16:10.193 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:16:10.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.213+0000 7f9876579640 5 monclient: authenticate success, global_id 4361 2024-03-29T11:16:10.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.213+0000 7f9876579640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:16:10.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.213+0000 7f986effd640 1 -- 172.21.15.47:0/3972229726 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9868001bd0 con 0x7f98700af900 2024-03-29T11:16:10.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.213+0000 7f986effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:10.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.214+0000 7f986effd640 1 -- 172.21.15.47:0/3972229726 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f9868001fe0 con 0x7f98700af900 2024-03-29T11:16:10.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.214+0000 7f986effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:10.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.214+0000 7f986effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:16:10.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.214+0000 7f986effd640 10 monclient: dump: 2024-03-29T11:16:10.193 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:16:10.193 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:16:10.193 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:10.193 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:10.193 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:10.194 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:16:10.194 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:16:10.194 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:16:10.194 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:16:10.194 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:16:10.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.214+0000 7f9876579640 10 monclient: get_monmap_and_config success 2024-03-29T11:16:10.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.214+0000 7f9876579640 4 set_mon_vals no callback set 2024-03-29T11:16:10.196 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.215+0000 7f9876579640 10 monclient: shutdown 2024-03-29T11:16:10.196 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.215+0000 7f9876579640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:10.196 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.215+0000 7f9876579640 1 -- 172.21.15.47:0/3972229726 >> v1:172.21.15.47:6790/0 conn(0x7f98700af900 legacy=0x7f9870149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:10.196 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.215+0000 7f9876579640 1 -- 172.21.15.47:0/3972229726 shutdown_connections 2024-03-29T11:16:10.196 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.215+0000 7f9876579640 1 -- 172.21.15.47:0/3972229726 >> 172.21.15.47:0/3972229726 conn(0x7f98700b35e0 msgr2=0x7f98700b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:10.197 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.216+0000 7f9876579640 1 -- 172.21.15.47:0/3972229726 shutdown_connections 2024-03-29T11:16:10.197 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.216+0000 7f9876579640 1 -- 172.21.15.47:0/3972229726 wait complete. 2024-03-29T11:16:10.197 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.216+0000 7f9876579640 10 monclient: build_initial_monmap 2024-03-29T11:16:10.197 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.216+0000 7f9876579640 10 monclient: monmap: 2024-03-29T11:16:10.197 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:16:10.197 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:16:10.197 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:16:10.197 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:16:10.197 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:10.197 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:16:10.197 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:16:10.197 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:16:10.197 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:16:10.197 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:16:10.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.217+0000 7f9876579640 1 librados: starting msgr at 2024-03-29T11:16:10.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.217+0000 7f9876579640 1 librados: starting objecter 2024-03-29T11:16:10.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.217+0000 7f9876579640 1 Processor -- start 2024-03-29T11:16:10.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.217+0000 7f9876579640 1 -- start start 2024-03-29T11:16:10.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.217+0000 7f9876579640 1 librados: setting wanted keys 2024-03-29T11:16:10.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.217+0000 7f9876579640 1 librados: calling monclient init 2024-03-29T11:16:10.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.217+0000 7f9876579640 10 monclient: init 2024-03-29T11:16:10.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.218+0000 7f9876579640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:10.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.219+0000 7f9876579640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:16:10.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.219+0000 7f9876579640 10 monclient(hunting): picked mon.noname-a con 0x7f98700a9ac0 addr v1:172.21.15.47:6789/0 2024-03-29T11:16:10.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.219+0000 7f9876579640 10 monclient(hunting): picked mon.noname-c con 0x7f98700af900 addr v1:172.21.15.47:6790/0 2024-03-29T11:16:10.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.219+0000 7f986ffff640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f98700a9ac0 0x7f98701380d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:58402/0 (socket says 172.21.15.47:58402) 2024-03-29T11:16:10.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.219+0000 7f986ffff640 1 -- 172.21.15.47:0/4084677059 learned_addr learned my addr 172.21.15.47:0/4084677059 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:16:10.205 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.219+0000 7f9876579640 10 monclient(hunting): picked mon.noname-b con 0x7f987013ee90 addr v1:172.21.15.121:6789/0 2024-03-29T11:16:10.205 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.220+0000 7f9876579640 1 -- 172.21.15.47:0/4084677059 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98701d7130 con 0x7f98700a9ac0 2024-03-29T11:16:10.205 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.220+0000 7f986cff9640 10 client.?.objecter ms_handle_connect 0x7f98700af900 2024-03-29T11:16:10.205 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.220+0000 7f986cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:10.205 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.220+0000 7f986cff9640 10 client.?.objecter ms_handle_connect 0x7f98700a9ac0 2024-03-29T11:16:10.205 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.220+0000 7f986cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:10.205 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.220+0000 7f9876579640 1 -- 172.21.15.47:0/4084677059 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f987013a170 con 0x7f987013ee90 2024-03-29T11:16:10.205 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.220+0000 7f9876579640 1 -- 172.21.15.47:0/4084677059 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f987013a450 con 0x7f98700af900 2024-03-29T11:16:10.205 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.220+0000 7f9876579640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:10.205 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.220+0000 7f9876579640 10 monclient(hunting): authenticate will time out at 1013.408555s 2024-03-29T11:16:10.206 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.220+0000 7f986cff9640 10 client.?.objecter ms_handle_connect 0x7f987013ee90 2024-03-29T11:16:10.206 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.220+0000 7f986cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:10.206 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.221+0000 7f986cff9640 1 -- 172.21.15.47:0/4084677059 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2740826495 0 0) 0x7f987013a450 con 0x7f98700af900 2024-03-29T11:16:10.206 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.221+0000 7f986cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:16:10.206 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.221+0000 7f986cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:10.206 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.221+0000 7f986cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:10.206 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.221+0000 7f986cff9640 10 monclient(hunting): my global_id is 4364 2024-03-29T11:16:10.206 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.221+0000 7f986cff9640 1 -- 172.21.15.47:0/4084677059 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f985c004eb0 con 0x7f98700af900 2024-03-29T11:16:10.206 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.221+0000 7f986cff9640 1 -- 172.21.15.47:0/4084677059 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 942289478 0 0) 0x7f987013a170 con 0x7f987013ee90 2024-03-29T11:16:10.206 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.221+0000 7f986cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:16:10.206 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.221+0000 7f986cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:10.206 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.221+0000 7f986cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:10.206 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.221+0000 7f986cff9640 10 monclient(hunting): my global_id is 4348 2024-03-29T11:16:10.206 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.221+0000 7f986cff9640 1 -- 172.21.15.47:0/4084677059 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f985c0051d0 con 0x7f987013ee90 2024-03-29T11:16:10.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.221+0000 7f986cff9640 1 -- 172.21.15.47:0/4084677059 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2710961726 0 0) 0x7f98701d7130 con 0x7f98700a9ac0 2024-03-29T11:16:10.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.221+0000 7f986cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:16:10.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.221+0000 7f986cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:10.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.221+0000 7f986cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:10.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.221+0000 7f986cff9640 10 monclient(hunting): my global_id is 4347 2024-03-29T11:16:10.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.221+0000 7f986cff9640 1 -- 172.21.15.47:0/4084677059 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f985c005510 con 0x7f98700a9ac0 2024-03-29T11:16:10.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.222+0000 7f986cff9640 1 -- 172.21.15.47:0/4084677059 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3301346681 0 0) 0x7f985c005510 con 0x7f98700a9ac0 2024-03-29T11:16:10.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.222+0000 7f986cff9640 1 -- 172.21.15.47:0/4084677059 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f985c005980 con 0x7f98700a9ac0 2024-03-29T11:16:10.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.222+0000 7f986cff9640 1 -- 172.21.15.47:0/4084677059 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f9860001770 con 0x7f98700a9ac0 2024-03-29T11:16:10.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.222+0000 7f986cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:10.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.222+0000 7f986cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:16:10.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.222+0000 7f986cff9640 10 monclient(hunting): dump: 2024-03-29T11:16:10.207 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:16:10.207 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:16:10.207 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:10.207 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:10.208 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:10.208 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:16:10.208 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:16:10.208 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:16:10.208 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:16:10.208 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:16:10.208 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.222+0000 7f986cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:16:10.208 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.222+0000 7f986cff9640 1 -- 172.21.15.47:0/4084677059 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3457595444 0 0) 0x7f985c0051d0 con 0x7f987013ee90 2024-03-29T11:16:10.208 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.222+0000 7f986cff9640 1 -- 172.21.15.47:0/4084677059 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f985c0082f0 con 0x7f987013ee90 2024-03-29T11:16:10.208 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.222+0000 7f986cff9640 1 -- 172.21.15.47:0/4084677059 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f9868001ee0 con 0x7f987013ee90 2024-03-29T11:16:10.208 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.222+0000 7f986cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:10.208 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.222+0000 7f986cff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:16:10.208 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.222+0000 7f986cff9640 10 monclient(hunting): dump: 2024-03-29T11:16:10.209 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:16:10.209 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:16:10.209 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:10.209 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:10.209 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:10.209 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:16:10.209 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:16:10.209 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:16:10.209 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:16:10.209 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:16:10.209 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.223+0000 7f986cff9640 1 -- 172.21.15.47:0/4084677059 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3326864470 0 0) 0x7f985c004eb0 con 0x7f98700af900 2024-03-29T11:16:10.209 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.223+0000 7f986cff9640 1 -- 172.21.15.47:0/4084677059 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f985c0051d0 con 0x7f98700af900 2024-03-29T11:16:10.209 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.223+0000 7f986cff9640 1 -- 172.21.15.47:0/4084677059 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f9864001850 con 0x7f98700af900 2024-03-29T11:16:10.209 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.223+0000 7f986cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:10.209 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.223+0000 7f986cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:16:10.209 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.223+0000 7f986cff9640 10 monclient(hunting): dump: 2024-03-29T11:16:10.210 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:16:10.210 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:16:10.210 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:10.210 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:10.210 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:10.210 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:16:10.210 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:16:10.210 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:16:10.210 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:16:10.210 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:16:10.210 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.223+0000 7f986cff9640 1 -- 172.21.15.47:0/4084677059 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1748134824 0 0) 0x7f985c0082f0 con 0x7f987013ee90 2024-03-29T11:16:10.210 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.223+0000 7f986cff9640 1 -- 172.21.15.47:0/4084677059 >> v1:172.21.15.47:6790/0 conn(0x7f98700af900 legacy=0x7f9870141b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:10.210 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.223+0000 7f986cff9640 1 -- 172.21.15.47:0/4084677059 >> v1:172.21.15.47:6789/0 conn(0x7f98700a9ac0 legacy=0x7f98701380d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:10.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.223+0000 7f986cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:16:10.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.223+0000 7f986cff9640 1 monclient: found mon.b 2024-03-29T11:16:10.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.223+0000 7f986cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:10.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.223+0000 7f986cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:16:10.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.223+0000 7f986cff9640 1 -- 172.21.15.47:0/4084677059 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98701d6c10 con 0x7f987013ee90 2024-03-29T11:16:10.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.223+0000 7f986cff9640 10 monclient: _finish_auth 0 2024-03-29T11:16:10.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.223+0000 7f986cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:16:10.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.223+0000 7f986cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:10.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.223+0000 7f9876579640 5 monclient: authenticate success, global_id 4348 2024-03-29T11:16:10.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.223+0000 7f9876579640 10 monclient: _renew_subs 2024-03-29T11:16:10.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.223+0000 7f986cff9640 1 -- 172.21.15.47:0/4084677059 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9868002040 con 0x7f987013ee90 2024-03-29T11:16:10.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.223+0000 7f9876579640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:16:10.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.223+0000 7f9876579640 1 -- 172.21.15.47:0/4084677059 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98701344e0 con 0x7f987013ee90 2024-03-29T11:16:10.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.223+0000 7f986cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:10.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.223+0000 7f986cff9640 1 -- 172.21.15.47:0/4084677059 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f9868003e40 con 0x7f987013ee90 2024-03-29T11:16:10.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.223+0000 7f986cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:10.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.223+0000 7f986cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:16:10.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.223+0000 7f986cff9640 10 monclient: dump: 2024-03-29T11:16:10.212 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:16:10.212 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:16:10.212 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:10.212 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:10.212 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:10.212 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:16:10.212 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:16:10.212 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:16:10.212 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:16:10.212 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:16:10.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.223+0000 7f9876579640 10 client.4348.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:16:10.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.224+0000 7f9876579640 10 monclient: _renew_subs 2024-03-29T11:16:10.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.224+0000 7f9876579640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:16:10.213 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.224+0000 7f9876579640 1 -- 172.21.15.47:0/4084677059 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9870142270 con 0x7f987013ee90 2024-03-29T11:16:10.213 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.224+0000 7f9876579640 1 librados: init done 2024-03-29T11:16:10.213 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.224+0000 7f986e7fc640 4 set_mon_vals no callback set 2024-03-29T11:16:10.213 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.225+0000 7f9876579640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:10.213 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.225+0000 7f9876579640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:10.213 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.225+0000 7f9876579640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:16:10.213 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.225+0000 7f9876579640 1 -- 172.21.15.47:0/4084677059 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9870075f10 con 0x7f987013ee90 2024-03-29T11:16:10.213 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.226+0000 7f986cff9640 1 -- 172.21.15.47:0/4084677059 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7f9868019e10 con 0x7f987013ee90 2024-03-29T11:16:10.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.229+0000 7f986ffff640 10 monclient: get_auth_request con 0x7f985c06eaa0 auth_method 0 2024-03-29T11:16:10.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.229+0000 7f986cff9640 1 -- 172.21.15.47:0/4084677059 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7f9868083a30 con 0x7f987013ee90 2024-03-29T11:16:10.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.229+0000 7f986cff9640 10 client.4348.objecter ms_dispatch 0x7f9870000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:16:10.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.229+0000 7f986cff9640 3 client.4348.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:16:10.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.229+0000 7f986cff9640 3 client.4348.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:16:10.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.229+0000 7f986cff9640 20 client.4348.objecter dump_active .. 0 homeless 2024-03-29T11:16:10.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.229+0000 7f986cff9640 10 client.4348.objecter ms_handle_connect 0x7f985c06eaa0 2024-03-29T11:16:10.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.236+0000 7f986cff9640 1 -- 172.21.15.47:0/4084677059 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f986804f450 con 0x7f987013ee90 2024-03-29T11:16:10.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.236+0000 7f986cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:10.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.236+0000 7f986cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:16:10.522 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.544+0000 7f9876579640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:10.522 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.544+0000 7f9876579640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:10.522 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.544+0000 7f9876579640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:16:10.522 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.544+0000 7f9876579640 1 -- 172.21.15.47:0/4084677059 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9870142570 con 0x7f987013ee90 2024-03-29T11:16:10.523 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.545+0000 7f986cff9640 1 -- 172.21.15.47:0/4084677059 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7f9868051380 con 0x7f987013ee90 2024-03-29T11:16:10.523 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.545+0000 7f986cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:10.523 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.545+0000 7f986cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:16:10.523 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:16:10.523 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:16:10.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.547+0000 7f9876579640 10 librados: watch_flush enter 2024-03-29T11:16:10.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.547+0000 7f9876579640 10 librados: watch_flush exit 2024-03-29T11:16:10.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.548+0000 7f9876579640 20 client.4348.objecter shutdown clearing up homeless session... 2024-03-29T11:16:10.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.548+0000 7f9876579640 10 client.4348.objecter successfully canceled tick 2024-03-29T11:16:10.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.548+0000 7f9876579640 1 -- 172.21.15.47:0/4084677059 >> v1:172.21.15.121:6816/1964117370 conn(0x7f985c06eaa0 legacy=0x7f985c070f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:10.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.548+0000 7f9876579640 10 monclient: shutdown 2024-03-29T11:16:10.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.548+0000 7f9876579640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:10.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.548+0000 7f9876579640 1 -- 172.21.15.47:0/4084677059 >> v1:172.21.15.121:6789/0 conn(0x7f987013ee90 legacy=0x7f98701de5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:10.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.548+0000 7f9876579640 1 -- 172.21.15.47:0/4084677059 shutdown_connections 2024-03-29T11:16:10.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.548+0000 7f9876579640 1 -- 172.21.15.47:0/4084677059 >> 172.21.15.47:0/4084677059 conn(0x7f98700b35e0 msgr2=0x7f987013ac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:10.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.548+0000 7f9876579640 1 -- 172.21.15.47:0/4084677059 shutdown_connections 2024-03-29T11:16:10.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.548+0000 7f9876579640 1 -- 172.21.15.47:0/4084677059 wait complete. 2024-03-29T11:16:10.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:10.548+0000 7f9876579640 1 librados: shutdown 2024-03-29T11:16:10.540 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:16:17.346 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:16:17.499 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.521+0000 7f934f59e640 10 monclient: get_monmap_and_config 2024-03-29T11:16:17.499 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.521+0000 7f934f59e640 10 monclient: build_initial_monmap 2024-03-29T11:16:17.499 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.521+0000 7f934f59e640 10 monclient: monmap: 2024-03-29T11:16:17.499 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:16:17.499 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:16:17.499 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:16:17.522567+0000 2024-03-29T11:16:17.499 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:16:17.522567+0000 2024-03-29T11:16:17.499 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:17.499 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:16:17.499 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:16:17.500 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:16:17.500 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:16:17.500 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:16:17.500 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.522+0000 7f934f59e640 1 Processor -- start 2024-03-29T11:16:17.500 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.522+0000 7f934f59e640 1 -- start start 2024-03-29T11:16:17.501 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.522+0000 7f934f59e640 10 monclient: init 2024-03-29T11:16:17.501 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.523+0000 7f934f59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:17.501 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.523+0000 7f934f59e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:16:17.501 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.523+0000 7f934f59e640 10 monclient(hunting): picked mon.noname-b con 0x7f93500b4920 addr v1:172.21.15.47:6790/0 2024-03-29T11:16:17.501 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.523+0000 7f934f59e640 10 monclient(hunting): picked mon.noname-a con 0x7f93500b4200 addr v1:172.21.15.47:6789/0 2024-03-29T11:16:17.501 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.524+0000 7f934f59e640 10 monclient(hunting): picked mon.noname-c con 0x7f935013f9a0 addr v1:172.21.15.121:6789/0 2024-03-29T11:16:17.501 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.524+0000 7f934f59e640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93500b0930 con 0x7f93500b4200 2024-03-29T11:16:17.501 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.524+0000 7f934f59e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93500af460 con 0x7f935013f9a0 2024-03-29T11:16:17.501 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.524+0000 7f934f59e640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93500aeda0 con 0x7f93500b4920 2024-03-29T11:16:17.501 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.524+0000 7f934f59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:17.501 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.524+0000 7f934f59e640 10 monclient(hunting): authenticate will time out at 1020.711996s 2024-03-29T11:16:17.502 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.524+0000 7f934e59c640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f93500b4920 0x7f935013f290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:53640/0 (socket says 172.21.15.47:53640) 2024-03-29T11:16:17.502 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.524+0000 7f934e59c640 1 -- 172.21.15.47:0/587122381 learned_addr learned my addr 172.21.15.47:0/587122381 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:16:17.502 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.524+0000 7f934d59a640 1 -- 172.21.15.47:0/587122381 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4078521098 0 0) 0x7f93500aeda0 con 0x7f93500b4920 2024-03-29T11:16:17.502 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.524+0000 7f934d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:16:17.502 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.524+0000 7f934d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:17.502 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.524+0000 7f934d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:17.503 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.524+0000 7f934d59a640 10 monclient(hunting): my global_id is 4367 2024-03-29T11:16:17.503 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.524+0000 7f934d59a640 1 -- 172.21.15.47:0/587122381 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9338003520 con 0x7f93500b4920 2024-03-29T11:16:17.503 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.525+0000 7f934d59a640 1 -- 172.21.15.47:0/587122381 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 402637571 0 0) 0x7f9338003520 con 0x7f93500b4920 2024-03-29T11:16:17.503 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.525+0000 7f934d59a640 1 -- 172.21.15.47:0/587122381 >> v1:172.21.15.121:6789/0 conn(0x7f935013f9a0 legacy=0x7f935014d8a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:16:17.504 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.527+0000 7f934d59a640 1 -- 172.21.15.47:0/587122381 >> v1:172.21.15.47:6789/0 conn(0x7f93500b4200 legacy=0x7f93500b0fc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:16:17.504 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.527+0000 7f934d59a640 10 monclient: _finish_hunting 0 2024-03-29T11:16:17.504 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.527+0000 7f934d59a640 1 monclient: found mon.noname-b 2024-03-29T11:16:17.505 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.527+0000 7f934d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:17.505 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.527+0000 7f934d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-29T11:16:17.505 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.527+0000 7f934d59a640 1 -- 172.21.15.47:0/587122381 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93500af980 con 0x7f93500b4920 2024-03-29T11:16:17.505 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.527+0000 7f934d59a640 10 monclient: _finish_auth 0 2024-03-29T11:16:17.505 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.527+0000 7f934d59a640 10 monclient: _check_auth_tickets 2024-03-29T11:16:17.505 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.527+0000 7f934d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:17.505 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.527+0000 7f934d59a640 1 -- 172.21.15.47:0/587122381 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f9340001690 con 0x7f93500b4920 2024-03-29T11:16:17.505 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.527+0000 7f934d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:17.505 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.527+0000 7f934d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:16:17.505 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.527+0000 7f934d59a640 10 monclient: dump: 2024-03-29T11:16:17.505 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:16:17.505 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:16:17.505 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:17.505 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:17.506 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:17.506 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:16:17.506 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:16:17.506 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:16:17.506 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:16:17.506 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:16:17.506 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.527+0000 7f934f59e640 5 monclient: authenticate success, global_id 4367 2024-03-29T11:16:17.506 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.527+0000 7f934f59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:16:17.507 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.527+0000 7f934d59a640 1 -- 172.21.15.47:0/587122381 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9340001920 con 0x7f93500b4920 2024-03-29T11:16:17.507 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.527+0000 7f934d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:17.507 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.527+0000 7f934d59a640 1 -- 172.21.15.47:0/587122381 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f9340001cb0 con 0x7f93500b4920 2024-03-29T11:16:17.507 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.527+0000 7f934d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:17.507 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.527+0000 7f934d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:16:17.507 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.527+0000 7f934d59a640 10 monclient: dump: 2024-03-29T11:16:17.507 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:16:17.507 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:16:17.508 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:17.508 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:17.508 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:17.508 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:16:17.508 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:16:17.508 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:16:17.508 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:16:17.508 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:16:17.508 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.527+0000 7f934f59e640 10 monclient: get_monmap_and_config success 2024-03-29T11:16:17.508 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.527+0000 7f934f59e640 4 set_mon_vals no callback set 2024-03-29T11:16:17.508 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.528+0000 7f934f59e640 10 monclient: shutdown 2024-03-29T11:16:17.508 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.528+0000 7f934f59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:17.508 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.528+0000 7f934f59e640 1 -- 172.21.15.47:0/587122381 >> v1:172.21.15.47:6790/0 conn(0x7f93500b4920 legacy=0x7f935013f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:17.508 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.529+0000 7f934f59e640 1 -- 172.21.15.47:0/587122381 shutdown_connections 2024-03-29T11:16:17.509 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.529+0000 7f934f59e640 1 -- 172.21.15.47:0/587122381 >> 172.21.15.47:0/587122381 conn(0x7f935006d230 msgr2=0x7f9350066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:17.509 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.529+0000 7f934f59e640 1 -- 172.21.15.47:0/587122381 shutdown_connections 2024-03-29T11:16:17.509 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.529+0000 7f934f59e640 1 -- 172.21.15.47:0/587122381 wait complete. 2024-03-29T11:16:17.509 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.530+0000 7f934f59e640 10 monclient: build_initial_monmap 2024-03-29T11:16:17.510 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.530+0000 7f934f59e640 10 monclient: monmap: 2024-03-29T11:16:17.510 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:16:17.510 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:16:17.510 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:16:17.510 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:16:17.510 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:17.510 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:16:17.510 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:16:17.510 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:16:17.510 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:16:17.510 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:16:17.510 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.531+0000 7f934f59e640 1 librados: starting msgr at 2024-03-29T11:16:17.510 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.531+0000 7f934f59e640 1 librados: starting objecter 2024-03-29T11:16:17.511 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.531+0000 7f934f59e640 1 Processor -- start 2024-03-29T11:16:17.511 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.532+0000 7f934f59e640 1 -- start start 2024-03-29T11:16:17.511 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.532+0000 7f934f59e640 1 librados: setting wanted keys 2024-03-29T11:16:17.511 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.532+0000 7f934f59e640 1 librados: calling monclient init 2024-03-29T11:16:17.511 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.532+0000 7f934f59e640 10 monclient: init 2024-03-29T11:16:17.511 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.533+0000 7f934f59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:17.511 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.533+0000 7f934f59e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:16:17.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.535+0000 7f934f59e640 10 monclient(hunting): picked mon.noname-a con 0x7f93500b4200 addr v1:172.21.15.47:6789/0 2024-03-29T11:16:17.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.535+0000 7f934f59e640 10 monclient(hunting): picked mon.noname-b con 0x7f93500b4920 addr v1:172.21.15.121:6789/0 2024-03-29T11:16:17.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.535+0000 7f934ed9d640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f935013f9a0 0x7f93501ef490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:53958/0 (socket says 172.21.15.47:53958) 2024-03-29T11:16:17.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.535+0000 7f934ed9d640 1 -- 172.21.15.47:0/2195267276 learned_addr learned my addr 172.21.15.47:0/2195267276 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:16:17.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.535+0000 7f934f59e640 10 monclient(hunting): picked mon.noname-c con 0x7f935013f9a0 addr v1:172.21.15.47:6790/0 2024-03-29T11:16:17.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.535+0000 7f934f59e640 1 -- 172.21.15.47:0/2195267276 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9350145aa0 con 0x7f93500b4200 2024-03-29T11:16:17.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.535+0000 7f934f59e640 1 -- 172.21.15.47:0/2195267276 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9350144a50 con 0x7f93500b4920 2024-03-29T11:16:17.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.535+0000 7f934f59e640 1 -- 172.21.15.47:0/2195267276 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9350144be0 con 0x7f935013f9a0 2024-03-29T11:16:17.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.535+0000 7f934f59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:17.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.535+0000 7f934f59e640 10 monclient(hunting): authenticate will time out at 1020.723660s 2024-03-29T11:16:17.514 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.537+0000 7f933effd640 10 client.?.objecter ms_handle_connect 0x7f935013f9a0 2024-03-29T11:16:17.515 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.537+0000 7f933effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:17.515 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.537+0000 7f933effd640 1 -- 172.21.15.47:0/2195267276 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1943274811 0 0) 0x7f9350144be0 con 0x7f935013f9a0 2024-03-29T11:16:17.515 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.537+0000 7f933effd640 10 monclient(hunting): _negotiate 2024-03-29T11:16:17.515 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.537+0000 7f933effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:17.515 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.537+0000 7f933effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:17.515 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.537+0000 7f933effd640 10 monclient(hunting): my global_id is 4370 2024-03-29T11:16:17.515 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.537+0000 7f933effd640 1 -- 172.21.15.47:0/2195267276 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9330004eb0 con 0x7f935013f9a0 2024-03-29T11:16:17.515 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.538+0000 7f933effd640 1 -- 172.21.15.47:0/2195267276 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2211752077 0 0) 0x7f9330004eb0 con 0x7f935013f9a0 2024-03-29T11:16:17.516 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.538+0000 7f933effd640 1 -- 172.21.15.47:0/2195267276 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9350144be0 con 0x7f935013f9a0 2024-03-29T11:16:17.516 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.538+0000 7f933effd640 1 -- 172.21.15.47:0/2195267276 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f9348001ad0 con 0x7f935013f9a0 2024-03-29T11:16:17.516 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.538+0000 7f933effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:17.516 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.538+0000 7f933effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:16:17.516 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.538+0000 7f933effd640 10 monclient(hunting): dump: 2024-03-29T11:16:17.516 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:16:17.516 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:16:17.516 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:17.516 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:17.516 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:17.516 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:16:17.516 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:16:17.516 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:16:17.516 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:16:17.517 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:16:17.517 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.538+0000 7f933effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:16:17.517 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.539+0000 7f933effd640 10 client.?.objecter ms_handle_connect 0x7f93500b4920 2024-03-29T11:16:17.518 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.539+0000 7f933effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:17.518 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.539+0000 7f933effd640 1 -- 172.21.15.47:0/2195267276 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3324608922 0 0) 0x7f9350144a50 con 0x7f93500b4920 2024-03-29T11:16:17.518 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.539+0000 7f933effd640 10 monclient(hunting): _negotiate 2024-03-29T11:16:17.518 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.539+0000 7f933effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:17.518 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.539+0000 7f933effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:17.518 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.540+0000 7f933effd640 10 monclient(hunting): my global_id is 4351 2024-03-29T11:16:17.518 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.540+0000 7f933effd640 1 -- 172.21.15.47:0/2195267276 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9330004eb0 con 0x7f93500b4920 2024-03-29T11:16:17.519 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.541+0000 7f933effd640 1 -- 172.21.15.47:0/2195267276 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1105620661 0 0) 0x7f9350144be0 con 0x7f935013f9a0 2024-03-29T11:16:17.519 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.541+0000 7f933effd640 1 -- 172.21.15.47:0/2195267276 >> v1:172.21.15.121:6789/0 conn(0x7f93500b4920 legacy=0x7f93501e7130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:17.519 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.541+0000 7f933effd640 1 -- 172.21.15.47:0/2195267276 >> v1:172.21.15.47:6789/0 conn(0x7f93500b4200 legacy=0x7f93501e0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:17.519 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.541+0000 7f933effd640 10 monclient: _finish_hunting 0 2024-03-29T11:16:17.519 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.541+0000 7f933effd640 1 monclient: found mon.c 2024-03-29T11:16:17.519 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.541+0000 7f933effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:17.519 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.541+0000 7f933effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:16:17.519 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.541+0000 7f933effd640 1 -- 172.21.15.47:0/2195267276 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9350144d70 con 0x7f935013f9a0 2024-03-29T11:16:17.519 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.541+0000 7f933effd640 10 monclient: _finish_auth 0 2024-03-29T11:16:17.519 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.541+0000 7f933effd640 10 monclient: _check_auth_tickets 2024-03-29T11:16:17.520 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.541+0000 7f933effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:17.520 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.541+0000 7f934f59e640 5 monclient: authenticate success, global_id 4370 2024-03-29T11:16:17.520 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.541+0000 7f934f59e640 10 monclient: _renew_subs 2024-03-29T11:16:17.520 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.541+0000 7f934f59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:16:17.520 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.541+0000 7f934f59e640 1 -- 172.21.15.47:0/2195267276 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93501e1560 con 0x7f935013f9a0 2024-03-29T11:16:17.520 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.541+0000 7f934f59e640 10 client.4370.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:16:17.520 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.541+0000 7f934f59e640 10 monclient: _renew_subs 2024-03-29T11:16:17.520 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.541+0000 7f934f59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:16:17.520 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.541+0000 7f934f59e640 1 -- 172.21.15.47:0/2195267276 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9350142320 con 0x7f935013f9a0 2024-03-29T11:16:17.520 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.541+0000 7f934f59e640 1 librados: init done 2024-03-29T11:16:17.520 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.542+0000 7f933effd640 1 -- 172.21.15.47:0/2195267276 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9348001c30 con 0x7f935013f9a0 2024-03-29T11:16:17.520 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.542+0000 7f933effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:17.521 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.542+0000 7f933ffff640 4 set_mon_vals no callback set 2024-03-29T11:16:17.521 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.542+0000 7f933effd640 1 -- 172.21.15.47:0/2195267276 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f9348004bb0 con 0x7f935013f9a0 2024-03-29T11:16:17.521 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.542+0000 7f933effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:17.521 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.542+0000 7f933effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:16:17.521 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.542+0000 7f933effd640 10 monclient: dump: 2024-03-29T11:16:17.521 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:16:17.521 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:16:17.521 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:17.521 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:17.521 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:17.521 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:16:17.521 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:16:17.521 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:16:17.521 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:16:17.522 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:16:17.524 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.543+0000 7f934f59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:17.524 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.543+0000 7f934f59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:17.524 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.544+0000 7f934f59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:16:17.524 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.544+0000 7f934f59e640 1 -- 172.21.15.47:0/2195267276 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9334005080 con 0x7f935013f9a0 2024-03-29T11:16:17.524 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.544+0000 7f933effd640 1 -- 172.21.15.47:0/2195267276 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7f934801ab80 con 0x7f935013f9a0 2024-03-29T11:16:17.524 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.545+0000 7f934e59c640 10 monclient: get_auth_request con 0x7f933006de00 auth_method 0 2024-03-29T11:16:17.524 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.545+0000 7f933effd640 1 -- 172.21.15.47:0/2195267276 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7f9348084be0 con 0x7f935013f9a0 2024-03-29T11:16:17.524 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.545+0000 7f933effd640 10 client.4370.objecter ms_dispatch 0x7f9350000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:16:17.524 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.545+0000 7f933effd640 3 client.4370.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:16:17.524 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.545+0000 7f933effd640 3 client.4370.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:16:17.524 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.545+0000 7f933effd640 20 client.4370.objecter dump_active .. 0 homeless 2024-03-29T11:16:17.524 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.545+0000 7f933effd640 10 client.4370.objecter ms_handle_connect 0x7f933006de00 2024-03-29T11:16:17.531 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.553+0000 7f933effd640 1 -- 172.21.15.47:0/2195267276 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f93480502f0 con 0x7f935013f9a0 2024-03-29T11:16:17.531 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.553+0000 7f933effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:17.531 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.553+0000 7f933effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:16:17.872 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.894+0000 7f934f59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:17.872 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.894+0000 7f934f59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:17.872 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.894+0000 7f934f59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:16:17.872 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.894+0000 7f934f59e640 1 -- 172.21.15.47:0/2195267276 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9334005330 con 0x7f935013f9a0 2024-03-29T11:16:17.873 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.896+0000 7f933effd640 1 -- 172.21.15.47:0/2195267276 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7f93480523f0 con 0x7f935013f9a0 2024-03-29T11:16:17.873 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.896+0000 7f933effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:17.873 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.896+0000 7f933effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:16:17.874 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:16:17.874 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:16:17.878 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.900+0000 7f934f59e640 10 librados: watch_flush enter 2024-03-29T11:16:17.878 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.900+0000 7f934f59e640 10 librados: watch_flush exit 2024-03-29T11:16:17.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.900+0000 7f934f59e640 20 client.4370.objecter shutdown clearing up homeless session... 2024-03-29T11:16:17.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.900+0000 7f934f59e640 10 client.4370.objecter successfully canceled tick 2024-03-29T11:16:17.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.900+0000 7f934f59e640 1 -- 172.21.15.47:0/2195267276 >> v1:172.21.15.121:6816/1964117370 conn(0x7f933006de00 legacy=0x7f93300702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:17.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.900+0000 7f934f59e640 10 monclient: shutdown 2024-03-29T11:16:17.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.900+0000 7f934f59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:17.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.900+0000 7f934f59e640 1 -- 172.21.15.47:0/2195267276 >> v1:172.21.15.47:6790/0 conn(0x7f935013f9a0 legacy=0x7f93501ef490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:17.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.901+0000 7f934f59e640 1 -- 172.21.15.47:0/2195267276 shutdown_connections 2024-03-29T11:16:17.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.901+0000 7f934f59e640 1 -- 172.21.15.47:0/2195267276 >> 172.21.15.47:0/2195267276 conn(0x7f935006d230 msgr2=0x7f935013b250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:17.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.901+0000 7f934f59e640 1 -- 172.21.15.47:0/2195267276 shutdown_connections 2024-03-29T11:16:17.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.901+0000 7f934f59e640 1 -- 172.21.15.47:0/2195267276 wait complete. 2024-03-29T11:16:17.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:17.901+0000 7f934f59e640 1 librados: shutdown 2024-03-29T11:16:17.895 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:16:24.703 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:16:24.850 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.872+0000 7f2c2c063640 10 monclient: get_monmap_and_config 2024-03-29T11:16:24.850 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.872+0000 7f2c2c063640 10 monclient: build_initial_monmap 2024-03-29T11:16:24.850 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.872+0000 7f2c2c063640 10 monclient: monmap: 2024-03-29T11:16:24.851 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:16:24.851 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:16:24.851 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:16:24.873422+0000 2024-03-29T11:16:24.851 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:16:24.873422+0000 2024-03-29T11:16:24.851 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:24.851 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:16:24.851 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:16:24.851 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:16:24.851 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:16:24.851 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:16:24.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.875+0000 7f2c2c063640 1 Processor -- start 2024-03-29T11:16:24.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.876+0000 7f2c2c063640 1 -- start start 2024-03-29T11:16:24.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.876+0000 7f2c2c063640 10 monclient: init 2024-03-29T11:16:24.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.876+0000 7f2c2c063640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:24.854 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.876+0000 7f2c2c063640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:16:24.854 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.876+0000 7f2c2c063640 10 monclient(hunting): picked mon.noname-c con 0x7f2c240b4710 addr v1:172.21.15.121:6789/0 2024-03-29T11:16:24.854 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.876+0000 7f2c2c063640 10 monclient(hunting): picked mon.noname-b con 0x7f2c241420f0 addr v1:172.21.15.47:6790/0 2024-03-29T11:16:24.854 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.876+0000 7f2c2c063640 10 monclient(hunting): picked mon.noname-a con 0x7f2c24138550 addr v1:172.21.15.47:6789/0 2024-03-29T11:16:24.854 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.876+0000 7f2c2c063640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c24070ae0 con 0x7f2c24138550 2024-03-29T11:16:24.854 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.876+0000 7f2c2c063640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c24070d50 con 0x7f2c240b4710 2024-03-29T11:16:24.854 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.876+0000 7f2c2c063640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c2413bfa0 con 0x7f2c241420f0 2024-03-29T11:16:24.854 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.876+0000 7f2c2c063640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:24.855 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.876+0000 7f2c2c063640 10 monclient(hunting): authenticate will time out at 1028.064875s 2024-03-29T11:16:24.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.879+0000 7f2c295fe640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f2c241420f0 0x7f2c24137e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:57830/0 (socket says 172.21.15.47:57830) 2024-03-29T11:16:24.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.879+0000 7f2c295fe640 1 -- 172.21.15.47:0/371866370 learned_addr learned my addr 172.21.15.47:0/371866370 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:16:24.857 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.880+0000 7f2c28dfd640 1 -- 172.21.15.47:0/371866370 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1707855706 0 0) 0x7f2c2413bfa0 con 0x7f2c241420f0 2024-03-29T11:16:24.857 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.880+0000 7f2c28dfd640 10 monclient(hunting): _negotiate 2024-03-29T11:16:24.857 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.880+0000 7f2c28dfd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:24.857 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.880+0000 7f2c28dfd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:24.857 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.880+0000 7f2c28dfd640 10 monclient(hunting): my global_id is 4373 2024-03-29T11:16:24.857 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.880+0000 7f2c28dfd640 1 -- 172.21.15.47:0/371866370 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c10003520 con 0x7f2c241420f0 2024-03-29T11:16:24.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.880+0000 7f2c28dfd640 1 -- 172.21.15.47:0/371866370 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1245472344 0 0) 0x7f2c10003520 con 0x7f2c241420f0 2024-03-29T11:16:24.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.880+0000 7f2c28dfd640 1 -- 172.21.15.47:0/371866370 >> v1:172.21.15.121:6789/0 conn(0x7f2c240b4710 legacy=0x7f2c241419e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:16:24.861 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.883+0000 7f2c28dfd640 1 -- 172.21.15.47:0/371866370 >> v1:172.21.15.47:6789/0 conn(0x7f2c24138550 legacy=0x7f2c24149d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:24.861 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.883+0000 7f2c28dfd640 10 monclient: _finish_hunting 0 2024-03-29T11:16:24.861 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.883+0000 7f2c28dfd640 1 monclient: found mon.noname-b 2024-03-29T11:16:24.861 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.883+0000 7f2c28dfd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:24.861 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.883+0000 7f2c28dfd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-29T11:16:24.861 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.883+0000 7f2c28dfd640 1 -- 172.21.15.47:0/371866370 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c2406f310 con 0x7f2c241420f0 2024-03-29T11:16:24.861 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.883+0000 7f2c28dfd640 10 monclient: _finish_auth 0 2024-03-29T11:16:24.861 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.883+0000 7f2c28dfd640 10 monclient: _check_auth_tickets 2024-03-29T11:16:24.861 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.883+0000 7f2c28dfd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:24.861 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.883+0000 7f2c28dfd640 1 -- 172.21.15.47:0/371866370 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f2c14001670 con 0x7f2c241420f0 2024-03-29T11:16:24.861 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.883+0000 7f2c28dfd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:24.862 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.883+0000 7f2c28dfd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:16:24.862 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.883+0000 7f2c28dfd640 10 monclient: dump: 2024-03-29T11:16:24.862 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:16:24.862 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:16:24.862 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:24.862 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:24.862 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:24.862 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:16:24.862 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:16:24.862 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:16:24.862 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:16:24.862 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:16:24.862 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.884+0000 7f2c2c063640 5 monclient: authenticate success, global_id 4373 2024-03-29T11:16:24.863 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.884+0000 7f2c2c063640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:16:24.863 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.884+0000 7f2c28dfd640 1 -- 172.21.15.47:0/371866370 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2c14001840 con 0x7f2c241420f0 2024-03-29T11:16:24.863 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.884+0000 7f2c28dfd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:24.863 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.884+0000 7f2c28dfd640 1 -- 172.21.15.47:0/371866370 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f2c14001c50 con 0x7f2c241420f0 2024-03-29T11:16:24.863 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.884+0000 7f2c28dfd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:24.863 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.884+0000 7f2c28dfd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:16:24.863 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.884+0000 7f2c28dfd640 10 monclient: dump: 2024-03-29T11:16:24.863 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:16:24.863 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:16:24.863 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:24.863 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:24.863 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:24.863 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:16:24.863 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:16:24.863 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:16:24.864 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:16:24.864 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:16:24.864 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.884+0000 7f2c2c063640 10 monclient: get_monmap_and_config success 2024-03-29T11:16:24.864 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.884+0000 7f2c2c063640 4 set_mon_vals no callback set 2024-03-29T11:16:24.864 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.885+0000 7f2c2c063640 10 monclient: shutdown 2024-03-29T11:16:24.864 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.885+0000 7f2c2c063640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:24.864 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.885+0000 7f2c2c063640 1 -- 172.21.15.47:0/371866370 >> v1:172.21.15.47:6790/0 conn(0x7f2c241420f0 legacy=0x7f2c24137e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:24.865 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.888+0000 7f2c2c063640 1 -- 172.21.15.47:0/371866370 shutdown_connections 2024-03-29T11:16:24.865 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.888+0000 7f2c2c063640 1 -- 172.21.15.47:0/371866370 >> 172.21.15.47:0/371866370 conn(0x7f2c24067180 msgr2=0x7f2c240b3c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:24.865 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.888+0000 7f2c2c063640 1 -- 172.21.15.47:0/371866370 shutdown_connections 2024-03-29T11:16:24.865 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.888+0000 7f2c2c063640 1 -- 172.21.15.47:0/371866370 wait complete. 2024-03-29T11:16:24.867 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.889+0000 7f2c2c063640 10 monclient: build_initial_monmap 2024-03-29T11:16:24.867 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.889+0000 7f2c2c063640 10 monclient: monmap: 2024-03-29T11:16:24.867 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:16:24.867 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:16:24.867 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:16:24.867 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:16:24.867 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:24.867 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:16:24.867 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:16:24.867 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:16:24.867 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:16:24.868 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:16:24.868 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.889+0000 7f2c2c063640 1 librados: starting msgr at 2024-03-29T11:16:24.868 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.889+0000 7f2c2c063640 1 librados: starting objecter 2024-03-29T11:16:24.868 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.889+0000 7f2c2c063640 1 Processor -- start 2024-03-29T11:16:24.869 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.890+0000 7f2c2c063640 1 -- start start 2024-03-29T11:16:24.870 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.890+0000 7f2c2c063640 1 librados: setting wanted keys 2024-03-29T11:16:24.870 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.890+0000 7f2c2c063640 1 librados: calling monclient init 2024-03-29T11:16:24.870 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.890+0000 7f2c2c063640 10 monclient: init 2024-03-29T11:16:24.870 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.890+0000 7f2c2c063640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:24.870 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.890+0000 7f2c2c063640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:16:24.870 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.890+0000 7f2c2c063640 10 monclient(hunting): picked mon.noname-b con 0x7f2c240b4710 addr v1:172.21.15.121:6789/0 2024-03-29T11:16:24.870 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.890+0000 7f2c2c063640 10 monclient(hunting): picked mon.noname-a con 0x7f2c24138550 addr v1:172.21.15.47:6789/0 2024-03-29T11:16:24.870 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.890+0000 7f2c2c063640 10 monclient(hunting): picked mon.noname-c con 0x7f2c241420f0 addr v1:172.21.15.47:6790/0 2024-03-29T11:16:24.870 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.890+0000 7f2c2c063640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c2413ec00 con 0x7f2c24138550 2024-03-29T11:16:24.870 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.890+0000 7f2c2c063640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c241d3110 con 0x7f2c240b4710 2024-03-29T11:16:24.870 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.890+0000 7f2c2c063640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c2413c260 con 0x7f2c241420f0 2024-03-29T11:16:24.870 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.890+0000 7f2c2c063640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:24.870 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.890+0000 7f2c2c063640 10 monclient(hunting): authenticate will time out at 1028.078790s 2024-03-29T11:16:24.870 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.891+0000 7f2c2a600640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f2c241420f0 0x7f2c2412fdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:58608/0 (socket says 172.21.15.47:58608) 2024-03-29T11:16:24.871 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.891+0000 7f2c2a600640 1 -- 172.21.15.47:0/1206332013 learned_addr learned my addr 172.21.15.47:0/1206332013 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:16:24.871 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.892+0000 7f2c1a7fc640 10 client.?.objecter ms_handle_connect 0x7f2c241420f0 2024-03-29T11:16:24.871 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.892+0000 7f2c1a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:24.871 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.892+0000 7f2c1a7fc640 1 -- 172.21.15.47:0/1206332013 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 722905635 0 0) 0x7f2c2413c260 con 0x7f2c241420f0 2024-03-29T11:16:24.871 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.892+0000 7f2c1a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:16:24.871 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.892+0000 7f2c1a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:24.872 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.892+0000 7f2c1a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:24.872 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.892+0000 7f2c1a7fc640 10 monclient(hunting): my global_id is 4376 2024-03-29T11:16:24.872 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.892+0000 7f2c1a7fc640 1 -- 172.21.15.47:0/1206332013 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c08004eb0 con 0x7f2c241420f0 2024-03-29T11:16:24.872 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.892+0000 7f2c1a7fc640 1 -- 172.21.15.47:0/1206332013 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3058807160 0 0) 0x7f2c08004eb0 con 0x7f2c241420f0 2024-03-29T11:16:24.872 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.892+0000 7f2c1a7fc640 1 -- 172.21.15.47:0/1206332013 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2c2413c260 con 0x7f2c241420f0 2024-03-29T11:16:24.872 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.892+0000 7f2c1a7fc640 1 -- 172.21.15.47:0/1206332013 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f2c20001990 con 0x7f2c241420f0 2024-03-29T11:16:24.872 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.892+0000 7f2c1a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:24.872 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.892+0000 7f2c1a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:16:24.872 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.892+0000 7f2c1a7fc640 10 monclient(hunting): dump: 2024-03-29T11:16:24.872 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:16:24.872 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:16:24.872 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:24.872 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:24.872 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:24.873 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:16:24.873 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:16:24.873 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:16:24.873 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:16:24.873 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:16:24.873 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.892+0000 7f2c1a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:16:24.873 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.893+0000 7f2c1a7fc640 1 -- 172.21.15.47:0/1206332013 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3304920499 0 0) 0x7f2c2413c260 con 0x7f2c241420f0 2024-03-29T11:16:24.873 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.893+0000 7f2c1a7fc640 1 -- 172.21.15.47:0/1206332013 >> v1:172.21.15.121:6789/0 conn(0x7f2c240b4710 legacy=0x7f2c2413d5e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:16:24.873 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.894+0000 7f2c1a7fc640 1 -- 172.21.15.47:0/1206332013 >> v1:172.21.15.47:6789/0 conn(0x7f2c24138550 legacy=0x7f2c241d3630 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:16:24.874 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.895+0000 7f2c1a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:16:24.874 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.895+0000 7f2c1a7fc640 1 monclient: found mon.c 2024-03-29T11:16:24.875 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.896+0000 7f2c1a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:24.875 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.896+0000 7f2c1a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:16:24.875 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.896+0000 7f2c1a7fc640 1 -- 172.21.15.47:0/1206332013 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c2413fa40 con 0x7f2c241420f0 2024-03-29T11:16:24.875 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.896+0000 7f2c1a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:16:24.875 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.896+0000 7f2c1a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:16:24.875 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.896+0000 7f2c1a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:24.875 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.896+0000 7f2c2c063640 5 monclient: authenticate success, global_id 4376 2024-03-29T11:16:24.875 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.896+0000 7f2c2c063640 10 monclient: _renew_subs 2024-03-29T11:16:24.875 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.896+0000 7f2c2c063640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:16:24.875 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.896+0000 7f2c2c063640 1 -- 172.21.15.47:0/1206332013 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c2413dcf0 con 0x7f2c241420f0 2024-03-29T11:16:24.875 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.896+0000 7f2c2c063640 10 client.4376.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:16:24.875 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.896+0000 7f2c2c063640 10 monclient: _renew_subs 2024-03-29T11:16:24.875 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.896+0000 7f2c2c063640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:16:24.875 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.896+0000 7f2c2c063640 1 -- 172.21.15.47:0/1206332013 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c24130bf0 con 0x7f2c241420f0 2024-03-29T11:16:24.876 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.896+0000 7f2c2c063640 1 librados: init done 2024-03-29T11:16:24.876 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.897+0000 7f2c1a7fc640 1 -- 172.21.15.47:0/1206332013 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2c20001af0 con 0x7f2c241420f0 2024-03-29T11:16:24.876 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.897+0000 7f2c1a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:24.876 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.897+0000 7f2c1a7fc640 1 -- 172.21.15.47:0/1206332013 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f2c20003a60 con 0x7f2c241420f0 2024-03-29T11:16:24.876 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.897+0000 7f2c1a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:24.876 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.897+0000 7f2c1a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:16:24.876 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.897+0000 7f2c1a7fc640 10 monclient: dump: 2024-03-29T11:16:24.876 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:16:24.876 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:16:24.876 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:24.876 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:24.876 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:24.876 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:16:24.876 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:16:24.877 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:16:24.877 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:16:24.877 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:16:24.878 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.898+0000 7f2c1bfff640 4 set_mon_vals no callback set 2024-03-29T11:16:24.878 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.898+0000 7f2c2c063640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:24.878 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.898+0000 7f2c2c063640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:24.878 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.899+0000 7f2c1a7fc640 1 -- 172.21.15.47:0/1206332013 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7f2c20003bc0 con 0x7f2c241420f0 2024-03-29T11:16:24.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.899+0000 7f2c2c063640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:16:24.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.899+0000 7f2c2c063640 1 -- 172.21.15.47:0/1206332013 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c0c005080 con 0x7f2c241420f0 2024-03-29T11:16:24.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.900+0000 7f2c29dff640 10 monclient: get_auth_request con 0x7f2c0806dc00 auth_method 0 2024-03-29T11:16:24.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.900+0000 7f2c1a7fc640 1 -- 172.21.15.47:0/1206332013 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7f2c20084b10 con 0x7f2c241420f0 2024-03-29T11:16:24.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.900+0000 7f2c1a7fc640 10 client.4376.objecter ms_dispatch 0x7f2c24000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:16:24.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.900+0000 7f2c1a7fc640 3 client.4376.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:16:24.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.900+0000 7f2c1a7fc640 3 client.4376.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:16:24.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.900+0000 7f2c1a7fc640 20 client.4376.objecter dump_active .. 0 homeless 2024-03-29T11:16:24.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.900+0000 7f2c1a7fc640 10 client.4376.objecter ms_handle_connect 0x7f2c0806dc00 2024-03-29T11:16:24.886 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.908+0000 7f2c1a7fc640 1 -- 172.21.15.47:0/1206332013 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2c200503f0 con 0x7f2c241420f0 2024-03-29T11:16:24.886 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.908+0000 7f2c1a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:24.886 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:24.908+0000 7f2c1a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:16:25.221 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:25.244+0000 7f2c2c063640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:25.221 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:25.244+0000 7f2c2c063640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:25.221 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:25.244+0000 7f2c2c063640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:16:25.221 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:25.244+0000 7f2c2c063640 1 -- 172.21.15.47:0/1206332013 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2c0c005330 con 0x7f2c241420f0 2024-03-29T11:16:25.222 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:25.244+0000 7f2c1a7fc640 1 -- 172.21.15.47:0/1206332013 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7f2c20052320 con 0x7f2c241420f0 2024-03-29T11:16:25.222 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:25.244+0000 7f2c1a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:25.222 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:25.244+0000 7f2c1a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:16:25.222 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:16:25.222 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:16:25.225 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:25.248+0000 7f2c2c063640 10 librados: watch_flush enter 2024-03-29T11:16:25.225 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:25.248+0000 7f2c2c063640 10 librados: watch_flush exit 2024-03-29T11:16:25.225 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:25.248+0000 7f2c2c063640 20 client.4376.objecter shutdown clearing up homeless session... 2024-03-29T11:16:25.226 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:25.248+0000 7f2c2c063640 10 client.4376.objecter successfully canceled tick 2024-03-29T11:16:25.227 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:25.249+0000 7f2c2c063640 1 -- 172.21.15.47:0/1206332013 >> v1:172.21.15.121:6816/1964117370 conn(0x7f2c0806dc00 legacy=0x7f2c080700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:25.227 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:25.249+0000 7f2c2c063640 10 monclient: shutdown 2024-03-29T11:16:25.227 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:25.249+0000 7f2c2c063640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:25.227 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:25.249+0000 7f2c2c063640 1 -- 172.21.15.47:0/1206332013 >> v1:172.21.15.47:6790/0 conn(0x7f2c241420f0 legacy=0x7f2c2412fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:25.227 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:25.249+0000 7f2c2c063640 1 -- 172.21.15.47:0/1206332013 shutdown_connections 2024-03-29T11:16:25.227 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:25.249+0000 7f2c2c063640 1 -- 172.21.15.47:0/1206332013 >> 172.21.15.47:0/1206332013 conn(0x7f2c24067180 msgr2=0x7f2c2406ddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:25.227 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:25.249+0000 7f2c2c063640 1 -- 172.21.15.47:0/1206332013 shutdown_connections 2024-03-29T11:16:25.227 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:25.249+0000 7f2c2c063640 1 -- 172.21.15.47:0/1206332013 wait complete. 2024-03-29T11:16:25.227 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:25.249+0000 7f2c2c063640 1 librados: shutdown 2024-03-29T11:16:25.247 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:16:32.054 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:16:32.204 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.226+0000 7fe0bbe8f640 10 monclient: get_monmap_and_config 2024-03-29T11:16:32.204 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.226+0000 7fe0bbe8f640 10 monclient: build_initial_monmap 2024-03-29T11:16:32.204 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.226+0000 7fe0bbe8f640 10 monclient: monmap: 2024-03-29T11:16:32.204 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:16:32.204 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:16:32.204 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:16:32.227288+0000 2024-03-29T11:16:32.204 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:16:32.227288+0000 2024-03-29T11:16:32.204 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:32.204 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:16:32.204 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:16:32.204 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:16:32.204 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:16:32.205 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:16:32.205 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.228+0000 7fe0bbe8f640 1 Processor -- start 2024-03-29T11:16:32.205 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.228+0000 7fe0bbe8f640 1 -- start start 2024-03-29T11:16:32.205 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.228+0000 7fe0bbe8f640 10 monclient: init 2024-03-29T11:16:32.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.229+0000 7fe0bbe8f640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:32.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.229+0000 7fe0bbe8f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:16:32.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.229+0000 7fe0b9c2b640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fe0b40b0860 0x7fe0b413e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:34782/0 (socket says 172.21.15.47:34782) 2024-03-29T11:16:32.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.229+0000 7fe0b9c2b640 1 -- 172.21.15.47:0/2486187735 learned_addr learned my addr 172.21.15.47:0/2486187735 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:16:32.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.229+0000 7fe0bbe8f640 10 monclient(hunting): picked mon.noname-b con 0x7fe0b40b0860 addr v1:172.21.15.47:6790/0 2024-03-29T11:16:32.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.229+0000 7fe0bbe8f640 10 monclient(hunting): picked mon.noname-a con 0x7fe0b406d9d0 addr v1:172.21.15.47:6789/0 2024-03-29T11:16:32.210 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.230+0000 7fe0bbe8f640 10 monclient(hunting): picked mon.noname-c con 0x7fe0b413ef30 addr v1:172.21.15.121:6789/0 2024-03-29T11:16:32.210 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.232+0000 7fe0bbe8f640 1 -- 172.21.15.47:0/2486187735 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0b40b0c40 con 0x7fe0b406d9d0 2024-03-29T11:16:32.210 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.232+0000 7fe0bbe8f640 1 -- 172.21.15.47:0/2486187735 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0b40aed10 con 0x7fe0b413ef30 2024-03-29T11:16:32.210 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.233+0000 7fe0bbe8f640 1 -- 172.21.15.47:0/2486187735 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0b40ab1b0 con 0x7fe0b40b0860 2024-03-29T11:16:32.210 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.233+0000 7fe0bbe8f640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:32.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.233+0000 7fe0bbe8f640 10 monclient(hunting): authenticate will time out at 1035.421103s 2024-03-29T11:16:32.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.233+0000 7fe0b8c29640 1 -- 172.21.15.47:0/2486187735 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1646305345 0 0) 0x7fe0b40aed10 con 0x7fe0b413ef30 2024-03-29T11:16:32.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.233+0000 7fe0b8c29640 10 monclient(hunting): _negotiate 2024-03-29T11:16:32.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.233+0000 7fe0b8c29640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:32.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.233+0000 7fe0b8c29640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:32.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.233+0000 7fe0b8c29640 10 monclient(hunting): my global_id is 4357 2024-03-29T11:16:32.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.233+0000 7fe0b8c29640 1 -- 172.21.15.47:0/2486187735 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0a0003520 con 0x7fe0b413ef30 2024-03-29T11:16:32.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.234+0000 7fe0b8c29640 1 -- 172.21.15.47:0/2486187735 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1461707240 0 0) 0x7fe0b40ab1b0 con 0x7fe0b40b0860 2024-03-29T11:16:32.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.234+0000 7fe0b8c29640 10 monclient(hunting): _negotiate 2024-03-29T11:16:32.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.234+0000 7fe0b8c29640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:32.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.234+0000 7fe0b8c29640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:32.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.234+0000 7fe0b8c29640 10 monclient(hunting): my global_id is 4379 2024-03-29T11:16:32.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.234+0000 7fe0b8c29640 1 -- 172.21.15.47:0/2486187735 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0a0003840 con 0x7fe0b40b0860 2024-03-29T11:16:32.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.234+0000 7fe0b8c29640 1 -- 172.21.15.47:0/2486187735 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1044450973 0 0) 0x7fe0a0003520 con 0x7fe0b413ef30 2024-03-29T11:16:32.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.234+0000 7fe0b8c29640 1 -- 172.21.15.47:0/2486187735 >> v1:172.21.15.47:6790/0 conn(0x7fe0b40b0860 legacy=0x7fe0b413e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:32.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.234+0000 7fe0b8c29640 1 -- 172.21.15.47:0/2486187735 >> v1:172.21.15.47:6789/0 conn(0x7fe0b406d9d0 legacy=0x7fe0b40af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:16:32.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.234+0000 7fe0b8c29640 10 monclient: _finish_hunting 0 2024-03-29T11:16:32.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.234+0000 7fe0b8c29640 1 monclient: found mon.noname-c 2024-03-29T11:16:32.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.234+0000 7fe0b8c29640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:32.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.234+0000 7fe0b8c29640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:16:32.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.234+0000 7fe0b8c29640 1 -- 172.21.15.47:0/2486187735 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0b40ab490 con 0x7fe0b413ef30 2024-03-29T11:16:32.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.234+0000 7fe0b8c29640 10 monclient: _finish_auth 0 2024-03-29T11:16:32.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.234+0000 7fe0b8c29640 10 monclient: _check_auth_tickets 2024-03-29T11:16:32.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.234+0000 7fe0b8c29640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:32.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.234+0000 7fe0b8c29640 1 -- 172.21.15.47:0/2486187735 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fe0ac0019f0 con 0x7fe0b413ef30 2024-03-29T11:16:32.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.234+0000 7fe0b8c29640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:32.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.234+0000 7fe0b8c29640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:16:32.216 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.234+0000 7fe0b8c29640 10 monclient: dump: 2024-03-29T11:16:32.216 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:16:32.216 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:16:32.216 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:32.216 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:32.216 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:32.216 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:16:32.216 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:16:32.216 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:16:32.216 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:16:32.216 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:16:32.216 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.234+0000 7fe0bbe8f640 5 monclient: authenticate success, global_id 4357 2024-03-29T11:16:32.216 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.234+0000 7fe0bbe8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:16:32.216 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.234+0000 7fe0b8c29640 1 -- 172.21.15.47:0/2486187735 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0ac001b50 con 0x7fe0b413ef30 2024-03-29T11:16:32.216 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.234+0000 7fe0b8c29640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:32.217 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.234+0000 7fe0b8c29640 1 -- 172.21.15.47:0/2486187735 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fe0ac001fa0 con 0x7fe0b413ef30 2024-03-29T11:16:32.217 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.234+0000 7fe0bbe8f640 10 monclient: get_monmap_and_config success 2024-03-29T11:16:32.217 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.234+0000 7fe0b8c29640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:32.217 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.235+0000 7fe0bbe8f640 4 set_mon_vals no callback set 2024-03-29T11:16:32.217 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.235+0000 7fe0b8c29640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:16:32.217 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.235+0000 7fe0b8c29640 10 monclient: dump: 2024-03-29T11:16:32.217 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:16:32.217 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:16:32.217 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:32.217 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:32.217 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:32.217 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:16:32.217 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:16:32.217 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:16:32.217 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:16:32.217 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:16:32.217 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.235+0000 7fe0bbe8f640 10 monclient: shutdown 2024-03-29T11:16:32.218 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.236+0000 7fe0bbe8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:32.218 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.236+0000 7fe0bbe8f640 1 -- 172.21.15.47:0/2486187735 >> v1:172.21.15.121:6789/0 conn(0x7fe0b413ef30 legacy=0x7fe0b4149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:32.218 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.236+0000 7fe0bbe8f640 1 -- 172.21.15.47:0/2486187735 shutdown_connections 2024-03-29T11:16:32.218 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.236+0000 7fe0bbe8f640 1 -- 172.21.15.47:0/2486187735 >> 172.21.15.47:0/2486187735 conn(0x7fe0b406b660 msgr2=0x7fe0b406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:32.218 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.236+0000 7fe0bbe8f640 1 -- 172.21.15.47:0/2486187735 shutdown_connections 2024-03-29T11:16:32.218 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.236+0000 7fe0bbe8f640 1 -- 172.21.15.47:0/2486187735 wait complete. 2024-03-29T11:16:32.220 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.238+0000 7fe0bbe8f640 10 monclient: build_initial_monmap 2024-03-29T11:16:32.220 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.238+0000 7fe0bbe8f640 10 monclient: monmap: 2024-03-29T11:16:32.220 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:16:32.220 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:16:32.221 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:16:32.221 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:16:32.221 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:32.221 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:16:32.221 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:16:32.221 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:16:32.221 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:16:32.221 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:16:32.221 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.238+0000 7fe0bbe8f640 1 librados: starting msgr at 2024-03-29T11:16:32.221 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.238+0000 7fe0bbe8f640 1 librados: starting objecter 2024-03-29T11:16:32.221 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.239+0000 7fe0bbe8f640 1 Processor -- start 2024-03-29T11:16:32.221 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.239+0000 7fe0bbe8f640 1 -- start start 2024-03-29T11:16:32.221 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.239+0000 7fe0bbe8f640 1 librados: setting wanted keys 2024-03-29T11:16:32.221 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.239+0000 7fe0bbe8f640 1 librados: calling monclient init 2024-03-29T11:16:32.221 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.239+0000 7fe0bbe8f640 10 monclient: init 2024-03-29T11:16:32.221 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.240+0000 7fe0bbe8f640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:32.222 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.240+0000 7fe0bbe8f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:16:32.222 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.242+0000 7fe0bbe8f640 10 monclient(hunting): picked mon.noname-b con 0x7fe0b406d9d0 addr v1:172.21.15.121:6789/0 2024-03-29T11:16:32.222 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.242+0000 7fe0b9c2b640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fe0b406d9d0 0x7fe0b40b2130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.47:38695/0 (socket says 172.21.15.47:38695) 2024-03-29T11:16:32.222 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.242+0000 7fe0b9c2b640 1 -- 172.21.15.47:0/2117336556 learned_addr learned my addr 172.21.15.47:0/2117336556 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:16:32.222 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.243+0000 7fe0bbe8f640 10 monclient(hunting): picked mon.noname-a con 0x7fe0b40b0860 addr v1:172.21.15.47:6789/0 2024-03-29T11:16:32.222 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.243+0000 7fe0bbe8f640 10 monclient(hunting): picked mon.noname-c con 0x7fe0b413ef30 addr v1:172.21.15.47:6790/0 2024-03-29T11:16:32.222 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.243+0000 7fe0bbe8f640 1 -- 172.21.15.47:0/2117336556 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0b4139e80 con 0x7fe0b40b0860 2024-03-29T11:16:32.222 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.243+0000 7fe0bbe8f640 1 -- 172.21.15.47:0/2117336556 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0b40b1670 con 0x7fe0b406d9d0 2024-03-29T11:16:32.222 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.243+0000 7fe0bbe8f640 1 -- 172.21.15.47:0/2117336556 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0b40b2840 con 0x7fe0b413ef30 2024-03-29T11:16:32.222 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.243+0000 7fe0bbe8f640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:32.222 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.243+0000 7fe0bbe8f640 10 monclient(hunting): authenticate will time out at 1035.431478s 2024-03-29T11:16:32.222 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.243+0000 7fe0aa7fc640 10 client.?.objecter ms_handle_connect 0x7fe0b406d9d0 2024-03-29T11:16:32.228 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.245+0000 7fe0aa7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:32.229 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.246+0000 7fe0aa7fc640 10 client.?.objecter ms_handle_connect 0x7fe0b413ef30 2024-03-29T11:16:32.229 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.246+0000 7fe0aa7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:32.229 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.246+0000 7fe0aa7fc640 10 client.?.objecter ms_handle_connect 0x7fe0b40b0860 2024-03-29T11:16:32.229 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.246+0000 7fe0aa7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:32.229 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.246+0000 7fe0aa7fc640 1 -- 172.21.15.47:0/2117336556 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4170908447 0 0) 0x7fe0b40b1670 con 0x7fe0b406d9d0 2024-03-29T11:16:32.229 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.246+0000 7fe0aa7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:16:32.229 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.246+0000 7fe0aa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:32.229 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.246+0000 7fe0aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:32.229 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.246+0000 7fe0aa7fc640 10 monclient(hunting): my global_id is 4360 2024-03-29T11:16:32.229 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.246+0000 7fe0aa7fc640 1 -- 172.21.15.47:0/2117336556 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe098004eb0 con 0x7fe0b406d9d0 2024-03-29T11:16:32.229 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.246+0000 7fe0aa7fc640 1 -- 172.21.15.47:0/2117336556 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2531542642 0 0) 0x7fe0b40b2840 con 0x7fe0b413ef30 2024-03-29T11:16:32.229 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.246+0000 7fe0aa7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:16:32.229 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.246+0000 7fe0aa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:32.230 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.246+0000 7fe0aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:32.230 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.246+0000 7fe0aa7fc640 10 monclient(hunting): my global_id is 4382 2024-03-29T11:16:32.230 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.246+0000 7fe0aa7fc640 1 -- 172.21.15.47:0/2117336556 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0b40b1670 con 0x7fe0b413ef30 2024-03-29T11:16:32.230 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.247+0000 7fe0aa7fc640 1 -- 172.21.15.47:0/2117336556 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2428614355 0 0) 0x7fe098004eb0 con 0x7fe0b406d9d0 2024-03-29T11:16:32.230 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.247+0000 7fe0aa7fc640 1 -- 172.21.15.47:0/2117336556 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0b40b2840 con 0x7fe0b406d9d0 2024-03-29T11:16:32.230 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.247+0000 7fe0aa7fc640 1 -- 172.21.15.47:0/2117336556 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fe0b00013b0 con 0x7fe0b406d9d0 2024-03-29T11:16:32.230 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.247+0000 7fe0aa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:32.230 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.247+0000 7fe0aa7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:16:32.230 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.247+0000 7fe0aa7fc640 10 monclient(hunting): dump: 2024-03-29T11:16:32.230 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:16:32.230 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:16:32.230 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:32.230 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:32.230 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:32.231 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:16:32.231 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:16:32.231 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:16:32.231 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:16:32.231 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:16:32.231 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.247+0000 7fe0aa7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:16:32.231 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.247+0000 7fe0aa7fc640 1 -- 172.21.15.47:0/2117336556 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4056317475 0 0) 0x7fe0b40b1670 con 0x7fe0b413ef30 2024-03-29T11:16:32.231 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.247+0000 7fe0aa7fc640 1 -- 172.21.15.47:0/2117336556 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe098004eb0 con 0x7fe0b413ef30 2024-03-29T11:16:32.231 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.247+0000 7fe0aa7fc640 1 -- 172.21.15.47:0/2117336556 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fe0ac0019b0 con 0x7fe0b413ef30 2024-03-29T11:16:32.231 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.247+0000 7fe0aa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:32.231 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.247+0000 7fe0aa7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:16:32.231 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.247+0000 7fe0aa7fc640 10 monclient(hunting): dump: 2024-03-29T11:16:32.231 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:16:32.231 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:16:32.231 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:32.232 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:32.232 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:32.232 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:16:32.232 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:16:32.232 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:16:32.232 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:16:32.232 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:16:32.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.247+0000 7fe0aa7fc640 1 -- 172.21.15.47:0/2117336556 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 610536542 0 0) 0x7fe0b40b2840 con 0x7fe0b406d9d0 2024-03-29T11:16:32.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.247+0000 7fe0aa7fc640 1 -- 172.21.15.47:0/2117336556 >> v1:172.21.15.47:6790/0 conn(0x7fe0b413ef30 legacy=0x7fe0b41ddb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:32.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.247+0000 7fe0aa7fc640 1 -- 172.21.15.47:0/2117336556 >> v1:172.21.15.47:6789/0 conn(0x7fe0b40b0860 legacy=0x7fe0b41d3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:32.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.247+0000 7fe0aa7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:16:32.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.247+0000 7fe0aa7fc640 1 monclient: found mon.b 2024-03-29T11:16:32.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.247+0000 7fe0aa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:32.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.247+0000 7fe0aa7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:16:32.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.247+0000 7fe0aa7fc640 1 -- 172.21.15.47:0/2117336556 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0b40b29d0 con 0x7fe0b406d9d0 2024-03-29T11:16:32.233 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.247+0000 7fe0aa7fc640 10 monclient: _finish_auth 0 2024-03-29T11:16:32.233 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.247+0000 7fe0aa7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:16:32.233 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.247+0000 7fe0aa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:32.233 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.247+0000 7fe0bbe8f640 5 monclient: authenticate success, global_id 4360 2024-03-29T11:16:32.233 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.247+0000 7fe0bbe8f640 10 monclient: _renew_subs 2024-03-29T11:16:32.233 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.247+0000 7fe0bbe8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:16:32.233 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.247+0000 7fe0bbe8f640 1 -- 172.21.15.47:0/2117336556 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0b40b4160 con 0x7fe0b406d9d0 2024-03-29T11:16:32.233 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.247+0000 7fe0bbe8f640 10 client.4360.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:16:32.233 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.247+0000 7fe0bbe8f640 10 monclient: _renew_subs 2024-03-29T11:16:32.233 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.247+0000 7fe0bbe8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:16:32.233 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.247+0000 7fe0bbe8f640 1 -- 172.21.15.47:0/2117336556 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0b41d3d80 con 0x7fe0b406d9d0 2024-03-29T11:16:32.233 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.247+0000 7fe0bbe8f640 1 librados: init done 2024-03-29T11:16:32.233 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.248+0000 7fe0aa7fc640 1 -- 172.21.15.47:0/2117336556 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0b0001880 con 0x7fe0b406d9d0 2024-03-29T11:16:32.233 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.248+0000 7fe0aa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:32.233 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.248+0000 7fe0aa7fc640 1 -- 172.21.15.47:0/2117336556 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fe0b0001c50 con 0x7fe0b406d9d0 2024-03-29T11:16:32.234 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.249+0000 7fe0ab7fe640 4 set_mon_vals no callback set 2024-03-29T11:16:32.234 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.249+0000 7fe0aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:32.234 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.249+0000 7fe0aa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:16:32.234 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.249+0000 7fe0aa7fc640 10 monclient: dump: 2024-03-29T11:16:32.234 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:16:32.234 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:16:32.234 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:32.234 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:32.234 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:32.234 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:16:32.234 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:16:32.234 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:16:32.234 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:16:32.234 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:16:32.234 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.249+0000 7fe0bbe8f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:32.235 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.249+0000 7fe0bbe8f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:32.235 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.250+0000 7fe0bbe8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:16:32.235 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.250+0000 7fe0bbe8f640 1 -- 172.21.15.47:0/2117336556 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe09c005080 con 0x7fe0b406d9d0 2024-03-29T11:16:32.235 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.251+0000 7fe0aa7fc640 1 -- 172.21.15.47:0/2117336556 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7fe0b001adc0 con 0x7fe0b406d9d0 2024-03-29T11:16:32.235 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.252+0000 7fe0aa7fc640 1 -- 172.21.15.47:0/2117336556 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7fe0b0084a50 con 0x7fe0b406d9d0 2024-03-29T11:16:32.236 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.252+0000 7fe0aa7fc640 10 client.4360.objecter ms_dispatch 0x7fe0b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:16:32.236 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.252+0000 7fe0aa7fc640 3 client.4360.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:16:32.236 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.252+0000 7fe0aa7fc640 3 client.4360.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:16:32.236 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.252+0000 7fe0aa7fc640 20 client.4360.objecter dump_active .. 0 homeless 2024-03-29T11:16:32.236 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.252+0000 7fe0b942a640 10 monclient: get_auth_request con 0x7fe09806df60 auth_method 0 2024-03-29T11:16:32.236 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.253+0000 7fe0aa7fc640 10 client.4360.objecter ms_handle_connect 0x7fe09806df60 2024-03-29T11:16:32.236 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.258+0000 7fe0aa7fc640 1 -- 172.21.15.47:0/2117336556 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe0b0050330 con 0x7fe0b406d9d0 2024-03-29T11:16:32.236 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.258+0000 7fe0aa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:32.236 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.258+0000 7fe0aa7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:16:32.571 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.592+0000 7fe0bbe8f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:32.572 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.592+0000 7fe0bbe8f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:32.572 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.592+0000 7fe0bbe8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:16:32.572 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.592+0000 7fe0bbe8f640 1 -- 172.21.15.47:0/2117336556 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe09c005330 con 0x7fe0b406d9d0 2024-03-29T11:16:32.572 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.594+0000 7fe0aa7fc640 1 -- 172.21.15.47:0/2117336556 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7fe0b0052260 con 0x7fe0b406d9d0 2024-03-29T11:16:32.572 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.594+0000 7fe0aa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:32.572 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.594+0000 7fe0aa7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:16:32.572 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:16:32.572 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:16:32.575 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.597+0000 7fe0bbe8f640 10 librados: watch_flush enter 2024-03-29T11:16:32.576 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.598+0000 7fe0bbe8f640 10 librados: watch_flush exit 2024-03-29T11:16:32.576 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.598+0000 7fe0bbe8f640 20 client.4360.objecter shutdown clearing up homeless session... 2024-03-29T11:16:32.576 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.598+0000 7fe0bbe8f640 10 client.4360.objecter successfully canceled tick 2024-03-29T11:16:32.576 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.598+0000 7fe0bbe8f640 1 -- 172.21.15.47:0/2117336556 >> v1:172.21.15.121:6816/1964117370 conn(0x7fe09806df60 legacy=0x7fe098070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:32.576 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.598+0000 7fe0bbe8f640 10 monclient: shutdown 2024-03-29T11:16:32.576 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.598+0000 7fe0bbe8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:32.576 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.598+0000 7fe0bbe8f640 1 -- 172.21.15.47:0/2117336556 >> v1:172.21.15.121:6789/0 conn(0x7fe0b406d9d0 legacy=0x7fe0b40b2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:32.576 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.598+0000 7fe0bbe8f640 1 -- 172.21.15.47:0/2117336556 shutdown_connections 2024-03-29T11:16:32.576 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.598+0000 7fe0bbe8f640 1 -- 172.21.15.47:0/2117336556 >> 172.21.15.47:0/2117336556 conn(0x7fe0b406b660 msgr2=0x7fe0b406ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:32.577 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.599+0000 7fe0bbe8f640 1 -- 172.21.15.47:0/2117336556 shutdown_connections 2024-03-29T11:16:32.577 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.599+0000 7fe0bbe8f640 1 -- 172.21.15.47:0/2117336556 wait complete. 2024-03-29T11:16:32.577 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:32.599+0000 7fe0bbe8f640 1 librados: shutdown 2024-03-29T11:16:32.594 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:16:39.406 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:16:39.555 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.577+0000 7f3f48bcf640 10 monclient: get_monmap_and_config 2024-03-29T11:16:39.555 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.577+0000 7f3f48bcf640 10 monclient: build_initial_monmap 2024-03-29T11:16:39.555 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.577+0000 7f3f48bcf640 10 monclient: monmap: 2024-03-29T11:16:39.555 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:16:39.555 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:16:39.556 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:16:39.579159+0000 2024-03-29T11:16:39.556 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:16:39.579159+0000 2024-03-29T11:16:39.556 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:39.556 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:16:39.556 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:16:39.556 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:16:39.556 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:16:39.556 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:16:39.556 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.578+0000 7f3f48bcf640 1 Processor -- start 2024-03-29T11:16:39.557 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.578+0000 7f3f48bcf640 1 -- start start 2024-03-29T11:16:39.557 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.578+0000 7f3f48bcf640 10 monclient: init 2024-03-29T11:16:39.560 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.579+0000 7f3f48bcf640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:39.560 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.579+0000 7f3f48bcf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:16:39.561 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.579+0000 7f3f48bcf640 10 monclient(hunting): picked mon.noname-a con 0x7f3f441457f0 addr v1:172.21.15.47:6789/0 2024-03-29T11:16:39.561 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.579+0000 7f3f48bcf640 10 monclient(hunting): picked mon.noname-c con 0x7f3f4413f650 addr v1:172.21.15.121:6789/0 2024-03-29T11:16:39.561 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.579+0000 7f3f48bcf640 10 monclient(hunting): picked mon.noname-b con 0x7f3f440a9d70 addr v1:172.21.15.47:6790/0 2024-03-29T11:16:39.561 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.579+0000 7f3f48bcf640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f440aaef0 con 0x7f3f441457f0 2024-03-29T11:16:39.561 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.582+0000 7f3f48bcf640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f44145bd0 con 0x7f3f4413f650 2024-03-29T11:16:39.561 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.582+0000 7f3f48bcf640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f44145eb0 con 0x7f3f440a9d70 2024-03-29T11:16:39.562 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.582+0000 7f3f48bcf640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:39.562 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.582+0000 7f3f48bcf640 10 monclient(hunting): authenticate will time out at 1042.771148s 2024-03-29T11:16:39.562 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.582+0000 7f3f41d9b640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f3f4413f650 0x7f3f440b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.47:48531/0 (socket says 172.21.15.47:48531) 2024-03-29T11:16:39.562 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.582+0000 7f3f41d9b640 1 -- 172.21.15.47:0/1002933025 learned_addr learned my addr 172.21.15.47:0/1002933025 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:16:39.562 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.583+0000 7f3f4159a640 1 -- 172.21.15.47:0/1002933025 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 625529411 0 0) 0x7f3f44145bd0 con 0x7f3f4413f650 2024-03-29T11:16:39.562 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.583+0000 7f3f4159a640 10 monclient(hunting): _negotiate 2024-03-29T11:16:39.562 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.583+0000 7f3f4159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:39.562 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.583+0000 7f3f4159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:39.562 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.583+0000 7f3f4159a640 10 monclient(hunting): my global_id is 4363 2024-03-29T11:16:39.562 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.583+0000 7f3f4159a640 1 -- 172.21.15.47:0/1002933025 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f2c003520 con 0x7f3f4413f650 2024-03-29T11:16:39.563 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.583+0000 7f3f4159a640 1 -- 172.21.15.47:0/1002933025 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2915457016 0 0) 0x7f3f2c003520 con 0x7f3f4413f650 2024-03-29T11:16:39.563 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.583+0000 7f3f4159a640 1 -- 172.21.15.47:0/1002933025 >> v1:172.21.15.47:6790/0 conn(0x7f3f440a9d70 legacy=0x7f3f4414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:39.563 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.583+0000 7f3f4159a640 1 -- 172.21.15.47:0/1002933025 >> v1:172.21.15.47:6789/0 conn(0x7f3f441457f0 legacy=0x7f3f4413ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:16:39.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.585+0000 7f3f4159a640 10 monclient: _finish_hunting 0 2024-03-29T11:16:39.566 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.585+0000 7f3f4159a640 1 monclient: found mon.noname-c 2024-03-29T11:16:39.566 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.585+0000 7f3f4159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:39.566 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.585+0000 7f3f4159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:16:39.566 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.585+0000 7f3f4159a640 1 -- 172.21.15.47:0/1002933025 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f44146340 con 0x7f3f4413f650 2024-03-29T11:16:39.566 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.585+0000 7f3f4159a640 10 monclient: _finish_auth 0 2024-03-29T11:16:39.566 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.585+0000 7f3f4159a640 10 monclient: _check_auth_tickets 2024-03-29T11:16:39.566 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.585+0000 7f3f4159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:39.566 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.585+0000 7f3f4159a640 1 -- 172.21.15.47:0/1002933025 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f3f38001690 con 0x7f3f4413f650 2024-03-29T11:16:39.566 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.585+0000 7f3f4159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:39.566 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.585+0000 7f3f4159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:16:39.566 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.585+0000 7f3f4159a640 10 monclient: dump: 2024-03-29T11:16:39.567 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:16:39.567 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:16:39.567 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:39.567 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:39.567 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:39.567 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:16:39.567 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:16:39.567 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:16:39.567 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:16:39.567 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:16:39.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.585+0000 7f3f48bcf640 5 monclient: authenticate success, global_id 4363 2024-03-29T11:16:39.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.585+0000 7f3f48bcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:16:39.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.585+0000 7f3f4159a640 1 -- 172.21.15.47:0/1002933025 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f38001920 con 0x7f3f4413f650 2024-03-29T11:16:39.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.585+0000 7f3f4159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:39.568 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.585+0000 7f3f48bcf640 10 monclient: get_monmap_and_config success 2024-03-29T11:16:39.568 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.585+0000 7f3f48bcf640 4 set_mon_vals no callback set 2024-03-29T11:16:39.568 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.586+0000 7f3f4159a640 1 -- 172.21.15.47:0/1002933025 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f3f38001d30 con 0x7f3f4413f650 2024-03-29T11:16:39.568 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.586+0000 7f3f4159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:39.568 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.586+0000 7f3f4159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:16:39.568 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.586+0000 7f3f4159a640 10 monclient: dump: 2024-03-29T11:16:39.568 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:16:39.568 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:16:39.568 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:39.568 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:39.568 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:39.568 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:16:39.568 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:16:39.568 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:16:39.568 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:16:39.569 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:16:39.569 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.586+0000 7f3f48bcf640 10 monclient: shutdown 2024-03-29T11:16:39.569 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.586+0000 7f3f48bcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:39.569 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.586+0000 7f3f48bcf640 1 -- 172.21.15.47:0/1002933025 >> v1:172.21.15.121:6789/0 conn(0x7f3f4413f650 legacy=0x7f3f440b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:39.569 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.586+0000 7f3f48bcf640 1 -- 172.21.15.47:0/1002933025 shutdown_connections 2024-03-29T11:16:39.569 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.586+0000 7f3f48bcf640 1 -- 172.21.15.47:0/1002933025 >> 172.21.15.47:0/1002933025 conn(0x7f3f44066a00 msgr2=0x7f3f44066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:39.569 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.587+0000 7f3f48bcf640 1 -- 172.21.15.47:0/1002933025 shutdown_connections 2024-03-29T11:16:39.569 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.587+0000 7f3f48bcf640 1 -- 172.21.15.47:0/1002933025 wait complete. 2024-03-29T11:16:39.570 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.588+0000 7f3f48bcf640 10 monclient: build_initial_monmap 2024-03-29T11:16:39.570 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.588+0000 7f3f48bcf640 10 monclient: monmap: 2024-03-29T11:16:39.570 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:16:39.570 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:16:39.570 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:16:39.570 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:16:39.570 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:39.571 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:16:39.571 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:16:39.571 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:16:39.571 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:16:39.571 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:16:39.571 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.588+0000 7f3f48bcf640 1 librados: starting msgr at 2024-03-29T11:16:39.571 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.588+0000 7f3f48bcf640 1 librados: starting objecter 2024-03-29T11:16:39.571 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.588+0000 7f3f48bcf640 1 Processor -- start 2024-03-29T11:16:39.571 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.588+0000 7f3f48bcf640 1 -- start start 2024-03-29T11:16:39.571 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.588+0000 7f3f48bcf640 1 librados: setting wanted keys 2024-03-29T11:16:39.571 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.588+0000 7f3f48bcf640 1 librados: calling monclient init 2024-03-29T11:16:39.571 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.588+0000 7f3f48bcf640 10 monclient: init 2024-03-29T11:16:39.571 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.589+0000 7f3f48bcf640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:39.571 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.589+0000 7f3f48bcf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:16:39.571 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.589+0000 7f3f48bcf640 10 monclient(hunting): picked mon.noname-a con 0x7f3f440a9d70 addr v1:172.21.15.47:6789/0 2024-03-29T11:16:39.572 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.589+0000 7f3f48bcf640 10 monclient(hunting): picked mon.noname-c con 0x7f3f4413f650 addr v1:172.21.15.47:6790/0 2024-03-29T11:16:39.572 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.589+0000 7f3f48bcf640 10 monclient(hunting): picked mon.noname-b con 0x7f3f441457f0 addr v1:172.21.15.121:6789/0 2024-03-29T11:16:39.572 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.589+0000 7f3f48bcf640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f441387a0 con 0x7f3f440a9d70 2024-03-29T11:16:39.572 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.589+0000 7f3f48bcf640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f440ad050 con 0x7f3f441457f0 2024-03-29T11:16:39.572 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.589+0000 7f3f48bcf640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f440ad330 con 0x7f3f4413f650 2024-03-29T11:16:39.572 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.589+0000 7f3f48bcf640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:39.572 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.589+0000 7f3f48bcf640 10 monclient(hunting): authenticate will time out at 1042.778679s 2024-03-29T11:16:39.574 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.595+0000 7f3f4259c640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f3f440a9d70 0x7f3f441d7390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:55691/0 (socket says 172.21.15.47:55691) 2024-03-29T11:16:39.574 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.595+0000 7f3f4259c640 1 -- 172.21.15.47:0/1245408930 learned_addr learned my addr 172.21.15.47:0/1245408930 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:16:39.574 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.595+0000 7f3f32ffd640 10 client.?.objecter ms_handle_connect 0x7f3f440a9d70 2024-03-29T11:16:39.574 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.595+0000 7f3f32ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:39.574 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.596+0000 7f3f32ffd640 10 client.?.objecter ms_handle_connect 0x7f3f441457f0 2024-03-29T11:16:39.574 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.596+0000 7f3f32ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:39.574 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.596+0000 7f3f32ffd640 1 -- 172.21.15.47:0/1245408930 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4200372833 0 0) 0x7f3f441387a0 con 0x7f3f440a9d70 2024-03-29T11:16:39.574 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.596+0000 7f3f32ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:16:39.574 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.596+0000 7f3f32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:39.574 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.596+0000 7f3f32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:39.574 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.596+0000 7f3f32ffd640 10 monclient(hunting): my global_id is 4356 2024-03-29T11:16:39.574 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.596+0000 7f3f32ffd640 1 -- 172.21.15.47:0/1245408930 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f24004eb0 con 0x7f3f440a9d70 2024-03-29T11:16:39.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.596+0000 7f3f32ffd640 1 -- 172.21.15.47:0/1245408930 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3989106552 0 0) 0x7f3f440ad050 con 0x7f3f441457f0 2024-03-29T11:16:39.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.596+0000 7f3f32ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:16:39.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.596+0000 7f3f32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:39.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.596+0000 7f3f32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:39.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.596+0000 7f3f32ffd640 10 monclient(hunting): my global_id is 4366 2024-03-29T11:16:39.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.596+0000 7f3f32ffd640 1 -- 172.21.15.47:0/1245408930 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f441387a0 con 0x7f3f441457f0 2024-03-29T11:16:39.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.596+0000 7f3f32ffd640 1 -- 172.21.15.47:0/1245408930 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3574528478 0 0) 0x7f3f24004eb0 con 0x7f3f440a9d70 2024-03-29T11:16:39.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.596+0000 7f3f32ffd640 1 -- 172.21.15.47:0/1245408930 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f440ad050 con 0x7f3f440a9d70 2024-03-29T11:16:39.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.596+0000 7f3f32ffd640 1 -- 172.21.15.47:0/1245408930 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f3f340017d0 con 0x7f3f440a9d70 2024-03-29T11:16:39.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.596+0000 7f3f32ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:39.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.596+0000 7f3f32ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:16:39.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.596+0000 7f3f32ffd640 10 monclient(hunting): dump: 2024-03-29T11:16:39.579 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:16:39.579 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:16:39.579 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:39.579 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:39.579 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:39.579 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:16:39.579 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:16:39.579 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:16:39.579 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:16:39.579 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:16:39.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.596+0000 7f3f32ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:16:39.580 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.596+0000 7f3f32ffd640 1 -- 172.21.15.47:0/1245408930 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3205450204 0 0) 0x7f3f441387a0 con 0x7f3f441457f0 2024-03-29T11:16:39.580 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.596+0000 7f3f32ffd640 1 -- 172.21.15.47:0/1245408930 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f24004eb0 con 0x7f3f441457f0 2024-03-29T11:16:39.580 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.596+0000 7f3f32ffd640 1 -- 172.21.15.47:0/1245408930 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f3f3c001ba0 con 0x7f3f441457f0 2024-03-29T11:16:39.580 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.596+0000 7f3f32ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:39.580 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.596+0000 7f3f32ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:16:39.580 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.596+0000 7f3f32ffd640 10 monclient(hunting): dump: 2024-03-29T11:16:39.580 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:16:39.580 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:16:39.580 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:39.580 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:39.580 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:39.580 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:16:39.580 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:16:39.580 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:16:39.580 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:16:39.581 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:16:39.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.596+0000 7f3f32ffd640 1 -- 172.21.15.47:0/1245408930 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1901074618 0 0) 0x7f3f440ad050 con 0x7f3f440a9d70 2024-03-29T11:16:39.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.596+0000 7f3f32ffd640 1 -- 172.21.15.47:0/1245408930 >> v1:172.21.15.47:6790/0 conn(0x7f3f4413f650 legacy=0x7f3f44137980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:16:39.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.597+0000 7f3f32ffd640 1 -- 172.21.15.47:0/1245408930 >> v1:172.21.15.121:6789/0 conn(0x7f3f441457f0 legacy=0x7f3f44138090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:39.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.597+0000 7f3f32ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:16:39.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.597+0000 7f3f32ffd640 1 monclient: found mon.a 2024-03-29T11:16:39.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.597+0000 7f3f32ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:39.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.597+0000 7f3f32ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:16:39.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.597+0000 7f3f32ffd640 1 -- 172.21.15.47:0/1245408930 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f440ac230 con 0x7f3f440a9d70 2024-03-29T11:16:39.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.597+0000 7f3f32ffd640 10 monclient: _finish_auth 0 2024-03-29T11:16:39.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.597+0000 7f3f32ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:16:39.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.597+0000 7f3f32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:39.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.597+0000 7f3f48bcf640 5 monclient: authenticate success, global_id 4356 2024-03-29T11:16:39.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.597+0000 7f3f48bcf640 10 monclient: _renew_subs 2024-03-29T11:16:39.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.597+0000 7f3f48bcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:16:39.582 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.597+0000 7f3f48bcf640 1 -- 172.21.15.47:0/1245408930 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f44142920 con 0x7f3f440a9d70 2024-03-29T11:16:39.582 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.597+0000 7f3f48bcf640 10 client.4356.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:16:39.582 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.597+0000 7f3f48bcf640 10 monclient: _renew_subs 2024-03-29T11:16:39.582 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.597+0000 7f3f48bcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:16:39.582 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.597+0000 7f3f48bcf640 1 -- 172.21.15.47:0/1245408930 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f441d7aa0 con 0x7f3f440a9d70 2024-03-29T11:16:39.582 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.597+0000 7f3f48bcf640 1 librados: init done 2024-03-29T11:16:39.582 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.597+0000 7f3f32ffd640 1 -- 172.21.15.47:0/1245408930 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f34001930 con 0x7f3f440a9d70 2024-03-29T11:16:39.582 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.597+0000 7f3f32ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:39.582 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.598+0000 7f3f32ffd640 1 -- 172.21.15.47:0/1245408930 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f3f34004a00 con 0x7f3f440a9d70 2024-03-29T11:16:39.582 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.598+0000 7f3f32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:39.582 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.598+0000 7f3f32ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:16:39.582 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.598+0000 7f3f32ffd640 10 monclient: dump: 2024-03-29T11:16:39.582 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:16:39.582 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:16:39.583 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:39.583 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:39.583 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:39.583 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:16:39.583 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:16:39.583 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:16:39.583 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:16:39.583 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:16:39.583 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.598+0000 7f3f33fff640 4 set_mon_vals no callback set 2024-03-29T11:16:39.583 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.599+0000 7f3f48bcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:39.583 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.599+0000 7f3f48bcf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:39.583 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.599+0000 7f3f48bcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:16:39.583 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.599+0000 7f3f32ffd640 1 -- 172.21.15.47:0/1245408930 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7f3f3401a9d0 con 0x7f3f440a9d70 2024-03-29T11:16:39.585 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.599+0000 7f3f48bcf640 1 -- 172.21.15.47:0/1245408930 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f28005080 con 0x7f3f440a9d70 2024-03-29T11:16:39.585 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.600+0000 7f3f32ffd640 1 -- 172.21.15.47:0/1245408930 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7f3f34084960 con 0x7f3f440a9d70 2024-03-29T11:16:39.585 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.600+0000 7f3f32ffd640 10 client.4356.objecter ms_dispatch 0x7f3f44000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:16:39.585 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.600+0000 7f3f32ffd640 3 client.4356.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:16:39.585 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.600+0000 7f3f32ffd640 3 client.4356.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:16:39.585 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.600+0000 7f3f32ffd640 20 client.4356.objecter dump_active .. 0 homeless 2024-03-29T11:16:39.585 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.600+0000 7f3f41d9b640 10 monclient: get_auth_request con 0x7f3f2406ddf0 auth_method 0 2024-03-29T11:16:39.585 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.601+0000 7f3f32ffd640 10 client.4356.objecter ms_handle_connect 0x7f3f2406ddf0 2024-03-29T11:16:39.587 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.609+0000 7f3f32ffd640 1 -- 172.21.15.47:0/1245408930 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3f34050240 con 0x7f3f440a9d70 2024-03-29T11:16:39.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.609+0000 7f3f32ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:39.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.609+0000 7f3f32ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:16:39.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.932+0000 7f3f48bcf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:39.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.932+0000 7f3f48bcf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:39.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.932+0000 7f3f48bcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:16:39.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.932+0000 7f3f48bcf640 1 -- 172.21.15.47:0/1245408930 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3f28005330 con 0x7f3f440a9d70 2024-03-29T11:16:39.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.934+0000 7f3f32ffd640 1 -- 172.21.15.47:0/1245408930 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7f3f34052170 con 0x7f3f440a9d70 2024-03-29T11:16:39.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.934+0000 7f3f32ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:39.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.934+0000 7f3f32ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:16:39.914 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:16:39.914 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:16:39.917 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.939+0000 7f3f48bcf640 10 librados: watch_flush enter 2024-03-29T11:16:39.917 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.939+0000 7f3f48bcf640 10 librados: watch_flush exit 2024-03-29T11:16:39.917 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.939+0000 7f3f48bcf640 20 client.4356.objecter shutdown clearing up homeless session... 2024-03-29T11:16:39.917 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.939+0000 7f3f48bcf640 10 client.4356.objecter successfully canceled tick 2024-03-29T11:16:39.917 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.939+0000 7f3f48bcf640 1 -- 172.21.15.47:0/1245408930 >> v1:172.21.15.121:6816/1964117370 conn(0x7f3f2406ddf0 legacy=0x7f3f240702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:39.917 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.939+0000 7f3f48bcf640 10 monclient: shutdown 2024-03-29T11:16:39.917 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.939+0000 7f3f48bcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:39.917 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.939+0000 7f3f48bcf640 1 -- 172.21.15.47:0/1245408930 >> v1:172.21.15.47:6789/0 conn(0x7f3f440a9d70 legacy=0x7f3f441d7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:39.922 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.944+0000 7f3f48bcf640 1 -- 172.21.15.47:0/1245408930 shutdown_connections 2024-03-29T11:16:39.922 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.944+0000 7f3f48bcf640 1 -- 172.21.15.47:0/1245408930 >> 172.21.15.47:0/1245408930 conn(0x7f3f44066a00 msgr2=0x7f3f440b4380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:39.922 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.944+0000 7f3f48bcf640 1 -- 172.21.15.47:0/1245408930 shutdown_connections 2024-03-29T11:16:39.922 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.944+0000 7f3f48bcf640 1 -- 172.21.15.47:0/1245408930 wait complete. 2024-03-29T11:16:39.922 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:39.944+0000 7f3f48bcf640 1 librados: shutdown 2024-03-29T11:16:39.939 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:16:46.747 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:16:46.897 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.919+0000 7fc4d5f82640 10 monclient: get_monmap_and_config 2024-03-29T11:16:46.897 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.919+0000 7fc4d5f82640 10 monclient: build_initial_monmap 2024-03-29T11:16:46.897 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.920+0000 7fc4d5f82640 10 monclient: monmap: 2024-03-29T11:16:46.897 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:16:46.897 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:16:46.897 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:16:46.920933+0000 2024-03-29T11:16:46.897 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:16:46.920933+0000 2024-03-29T11:16:46.897 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:46.898 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:16:46.898 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:16:46.898 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:16:46.898 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:16:46.898 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:16:46.902 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.922+0000 7fc4d5f82640 1 Processor -- start 2024-03-29T11:16:46.902 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.922+0000 7fc4d5f82640 1 -- start start 2024-03-29T11:16:46.902 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.922+0000 7fc4d5f82640 10 monclient: init 2024-03-29T11:16:46.902 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.923+0000 7fc4d5f82640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:46.902 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.923+0000 7fc4d5f82640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:16:46.902 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.923+0000 7fc4d5f82640 10 monclient(hunting): picked mon.noname-a con 0x7fc4d00b4330 addr v1:172.21.15.47:6789/0 2024-03-29T11:16:46.902 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.923+0000 7fc4d5f82640 10 monclient(hunting): picked mon.noname-b con 0x7fc4d01470b0 addr v1:172.21.15.47:6790/0 2024-03-29T11:16:46.902 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.923+0000 7fc4d5f82640 10 monclient(hunting): picked mon.noname-c con 0x7fc4d01439b0 addr v1:172.21.15.121:6789/0 2024-03-29T11:16:46.902 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.923+0000 7fc4d5f82640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4d006f2a0 con 0x7fc4d00b4330 2024-03-29T11:16:46.902 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.923+0000 7fc4d5f82640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4d0146b90 con 0x7fc4d01439b0 2024-03-29T11:16:46.902 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.923+0000 7fc4d5f82640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4d00b2ca0 con 0x7fc4d01470b0 2024-03-29T11:16:46.902 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.923+0000 7fc4d5f82640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:46.903 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.923+0000 7fc4d5f82640 10 monclient(hunting): authenticate will time out at 1050.111628s 2024-03-29T11:16:46.904 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.927+0000 7fc4cffff640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fc4d01439b0 0x7fc4d0151d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.47:33419/0 (socket says 172.21.15.47:33419) 2024-03-29T11:16:46.904 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.927+0000 7fc4cffff640 1 -- 172.21.15.47:0/1781980202 learned_addr learned my addr 172.21.15.47:0/1781980202 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:16:46.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.928+0000 7fc4ce7fc640 1 -- 172.21.15.47:0/1781980202 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2557761937 0 0) 0x7fc4d00b2ca0 con 0x7fc4d01470b0 2024-03-29T11:16:46.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.928+0000 7fc4ce7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:16:46.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.928+0000 7fc4ce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:46.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.928+0000 7fc4ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:46.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.928+0000 7fc4ce7fc640 10 monclient(hunting): my global_id is 4388 2024-03-29T11:16:46.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.928+0000 7fc4ce7fc640 1 -- 172.21.15.47:0/1781980202 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4b8003520 con 0x7fc4d01470b0 2024-03-29T11:16:46.908 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.930+0000 7fc4ce7fc640 1 -- 172.21.15.47:0/1781980202 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2733637550 0 0) 0x7fc4d0146b90 con 0x7fc4d01439b0 2024-03-29T11:16:46.908 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.930+0000 7fc4ce7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:16:46.908 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.930+0000 7fc4ce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:46.908 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.930+0000 7fc4ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:46.908 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.930+0000 7fc4ce7fc640 10 monclient(hunting): my global_id is 4369 2024-03-29T11:16:46.908 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.930+0000 7fc4ce7fc640 1 -- 172.21.15.47:0/1781980202 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4d00b2ca0 con 0x7fc4d01439b0 2024-03-29T11:16:46.909 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.930+0000 7fc4ce7fc640 1 -- 172.21.15.47:0/1781980202 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2696488654 0 0) 0x7fc4b8003520 con 0x7fc4d01470b0 2024-03-29T11:16:46.909 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.930+0000 7fc4ce7fc640 1 -- 172.21.15.47:0/1781980202 >> v1:172.21.15.121:6789/0 conn(0x7fc4d01439b0 legacy=0x7fc4d0151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:46.909 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.930+0000 7fc4ce7fc640 1 -- 172.21.15.47:0/1781980202 >> v1:172.21.15.47:6789/0 conn(0x7fc4d00b4330 legacy=0x7fc4d00b4710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:16:46.909 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.930+0000 7fc4ce7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:16:46.909 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.930+0000 7fc4ce7fc640 1 monclient: found mon.noname-b 2024-03-29T11:16:46.909 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.930+0000 7fc4ce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:46.909 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.930+0000 7fc4ce7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-29T11:16:46.909 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.930+0000 7fc4ce7fc640 1 -- 172.21.15.47:0/1781980202 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4d006e0c0 con 0x7fc4d01470b0 2024-03-29T11:16:46.909 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.930+0000 7fc4ce7fc640 10 monclient: _finish_auth 0 2024-03-29T11:16:46.909 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.930+0000 7fc4ce7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:16:46.909 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.930+0000 7fc4ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:46.909 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.930+0000 7fc4ce7fc640 1 -- 172.21.15.47:0/1781980202 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fc4c4001610 con 0x7fc4d01470b0 2024-03-29T11:16:46.909 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.930+0000 7fc4ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:46.909 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.930+0000 7fc4ce7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:16:46.910 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.930+0000 7fc4ce7fc640 10 monclient: dump: 2024-03-29T11:16:46.910 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:16:46.910 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:16:46.910 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:46.910 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:46.910 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:46.910 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:16:46.910 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:16:46.910 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:16:46.910 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:16:46.910 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:16:46.910 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.930+0000 7fc4d5f82640 5 monclient: authenticate success, global_id 4388 2024-03-29T11:16:46.910 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.930+0000 7fc4d5f82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:16:46.910 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.931+0000 7fc4ce7fc640 1 -- 172.21.15.47:0/1781980202 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc4c4001770 con 0x7fc4d01470b0 2024-03-29T11:16:46.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.931+0000 7fc4ce7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:46.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.931+0000 7fc4d5f82640 10 monclient: get_monmap_and_config success 2024-03-29T11:16:46.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.931+0000 7fc4d5f82640 4 set_mon_vals no callback set 2024-03-29T11:16:46.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.932+0000 7fc4ce7fc640 1 -- 172.21.15.47:0/1781980202 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fc4c4001b80 con 0x7fc4d01470b0 2024-03-29T11:16:46.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.932+0000 7fc4ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:46.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.932+0000 7fc4ce7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:16:46.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.932+0000 7fc4ce7fc640 10 monclient: dump: 2024-03-29T11:16:46.911 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:16:46.911 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:16:46.912 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:46.912 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:46.912 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:46.912 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:16:46.912 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:16:46.912 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:16:46.912 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:16:46.912 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:16:46.913 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.934+0000 7fc4d5f82640 10 monclient: shutdown 2024-03-29T11:16:46.913 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.934+0000 7fc4d5f82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:46.913 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.934+0000 7fc4d5f82640 1 -- 172.21.15.47:0/1781980202 >> v1:172.21.15.47:6790/0 conn(0x7fc4d01470b0 legacy=0x7fc4d00b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:46.913 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.936+0000 7fc4d5f82640 1 -- 172.21.15.47:0/1781980202 shutdown_connections 2024-03-29T11:16:46.913 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.936+0000 7fc4d5f82640 1 -- 172.21.15.47:0/1781980202 >> 172.21.15.47:0/1781980202 conn(0x7fc4d0066a00 msgr2=0x7fc4d0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:46.916 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.939+0000 7fc4d5f82640 1 -- 172.21.15.47:0/1781980202 shutdown_connections 2024-03-29T11:16:46.918 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.941+0000 7fc4d5f82640 1 -- 172.21.15.47:0/1781980202 wait complete. 2024-03-29T11:16:46.919 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.942+0000 7fc4d5f82640 10 monclient: build_initial_monmap 2024-03-29T11:16:46.919 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.942+0000 7fc4d5f82640 10 monclient: monmap: 2024-03-29T11:16:46.919 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:16:46.919 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:16:46.919 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:16:46.920 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:16:46.920 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:46.920 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:16:46.920 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:16:46.920 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:16:46.920 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:16:46.920 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:16:46.921 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.942+0000 7fc4d5f82640 1 librados: starting msgr at 2024-03-29T11:16:46.921 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.942+0000 7fc4d5f82640 1 librados: starting objecter 2024-03-29T11:16:46.921 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.943+0000 7fc4d5f82640 1 Processor -- start 2024-03-29T11:16:46.921 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.943+0000 7fc4d5f82640 1 -- start start 2024-03-29T11:16:46.921 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.943+0000 7fc4d5f82640 1 librados: setting wanted keys 2024-03-29T11:16:46.921 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.943+0000 7fc4d5f82640 1 librados: calling monclient init 2024-03-29T11:16:46.921 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.943+0000 7fc4d5f82640 10 monclient: init 2024-03-29T11:16:46.921 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.944+0000 7fc4d5f82640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:46.922 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.944+0000 7fc4d5f82640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:16:46.924 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.947+0000 7fc4d5f82640 10 monclient(hunting): picked mon.noname-a con 0x7fc4d00b4330 addr v1:172.21.15.47:6789/0 2024-03-29T11:16:46.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.948+0000 7fc4d5f82640 10 monclient(hunting): picked mon.noname-b con 0x7fc4d01439b0 addr v1:172.21.15.121:6789/0 2024-03-29T11:16:46.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.948+0000 7fc4d5f82640 10 monclient(hunting): picked mon.noname-c con 0x7fc4d01470b0 addr v1:172.21.15.47:6790/0 2024-03-29T11:16:46.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.948+0000 7fc4d5f82640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4d01db3d0 con 0x7fc4d00b4330 2024-03-29T11:16:46.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.948+0000 7fc4d5f82640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4d01df4e0 con 0x7fc4d01439b0 2024-03-29T11:16:46.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.948+0000 7fc4d5f82640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4d01df670 con 0x7fc4d01470b0 2024-03-29T11:16:46.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.948+0000 7fc4d5f82640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:46.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.948+0000 7fc4d5f82640 10 monclient(hunting): authenticate will time out at 1050.136532s 2024-03-29T11:16:46.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.951+0000 7fc4cf7fe640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fc4d00b4330 0x7fc4d01dedd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:42045/0 (socket says 172.21.15.47:42045) 2024-03-29T11:16:46.929 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.951+0000 7fc4cf7fe640 1 -- 172.21.15.47:0/2408031771 learned_addr learned my addr 172.21.15.47:0/2408031771 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:16:46.929 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.951+0000 7fc4b7fff640 10 client.?.objecter ms_handle_connect 0x7fc4d00b4330 2024-03-29T11:16:46.929 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.951+0000 7fc4b7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:46.936 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.953+0000 7fc4b7fff640 1 -- 172.21.15.47:0/2408031771 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3042212633 0 0) 0x7fc4d01db3d0 con 0x7fc4d00b4330 2024-03-29T11:16:46.936 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.954+0000 7fc4b7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:16:46.936 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.954+0000 7fc4b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:46.936 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.954+0000 7fc4b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:46.936 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.954+0000 7fc4b7fff640 10 monclient(hunting): my global_id is 4359 2024-03-29T11:16:46.936 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.954+0000 7fc4b7fff640 1 -- 172.21.15.47:0/2408031771 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4bc004eb0 con 0x7fc4d00b4330 2024-03-29T11:16:46.937 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.954+0000 7fc4b7fff640 10 client.?.objecter ms_handle_connect 0x7fc4d01439b0 2024-03-29T11:16:46.937 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.954+0000 7fc4b7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:46.937 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.954+0000 7fc4b7fff640 10 client.?.objecter ms_handle_connect 0x7fc4d01470b0 2024-03-29T11:16:46.937 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.954+0000 7fc4b7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:46.937 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.954+0000 7fc4b7fff640 1 -- 172.21.15.47:0/2408031771 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 581645754 0 0) 0x7fc4d01df4e0 con 0x7fc4d01439b0 2024-03-29T11:16:46.937 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.954+0000 7fc4b7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:16:46.937 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.954+0000 7fc4b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:46.937 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.954+0000 7fc4b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:46.937 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.955+0000 7fc4b7fff640 10 monclient(hunting): my global_id is 4372 2024-03-29T11:16:46.937 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.955+0000 7fc4b7fff640 1 -- 172.21.15.47:0/2408031771 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4d01db3d0 con 0x7fc4d01439b0 2024-03-29T11:16:46.937 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.955+0000 7fc4b7fff640 1 -- 172.21.15.47:0/2408031771 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1793079122 0 0) 0x7fc4d01df670 con 0x7fc4d01470b0 2024-03-29T11:16:46.937 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.955+0000 7fc4b7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:16:46.937 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.955+0000 7fc4b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:46.938 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.955+0000 7fc4b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:46.938 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.955+0000 7fc4b7fff640 10 monclient(hunting): my global_id is 4391 2024-03-29T11:16:46.938 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.955+0000 7fc4b7fff640 1 -- 172.21.15.47:0/2408031771 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4d01df4e0 con 0x7fc4d01470b0 2024-03-29T11:16:46.938 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.955+0000 7fc4b7fff640 1 -- 172.21.15.47:0/2408031771 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3611388191 0 0) 0x7fc4bc004eb0 con 0x7fc4d00b4330 2024-03-29T11:16:46.938 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.955+0000 7fc4b7fff640 1 -- 172.21.15.47:0/2408031771 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc4d01df670 con 0x7fc4d00b4330 2024-03-29T11:16:46.938 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.955+0000 7fc4b7fff640 1 -- 172.21.15.47:0/2408031771 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fc4c8001ad0 con 0x7fc4d00b4330 2024-03-29T11:16:46.938 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.955+0000 7fc4b7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:46.938 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.955+0000 7fc4b7fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:16:46.938 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.955+0000 7fc4b7fff640 10 monclient(hunting): dump: 2024-03-29T11:16:46.938 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:16:46.939 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:16:46.939 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:46.939 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:46.939 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:46.939 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:16:46.939 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:16:46.939 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:16:46.939 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:16:46.939 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:16:46.939 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.955+0000 7fc4b7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:16:46.939 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.956+0000 7fc4b7fff640 1 -- 172.21.15.47:0/2408031771 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 228419723 0 0) 0x7fc4d01df4e0 con 0x7fc4d01470b0 2024-03-29T11:16:46.939 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.956+0000 7fc4b7fff640 1 -- 172.21.15.47:0/2408031771 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc4bc004eb0 con 0x7fc4d01470b0 2024-03-29T11:16:46.939 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.956+0000 7fc4b7fff640 1 -- 172.21.15.47:0/2408031771 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1339737172 0 0) 0x7fc4d01df670 con 0x7fc4d00b4330 2024-03-29T11:16:46.939 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.956+0000 7fc4b7fff640 1 -- 172.21.15.47:0/2408031771 >> v1:172.21.15.47:6790/0 conn(0x7fc4d01470b0 legacy=0x7fc4d01e6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:46.940 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.956+0000 7fc4b7fff640 1 -- 172.21.15.47:0/2408031771 >> v1:172.21.15.121:6789/0 conn(0x7fc4d01439b0 legacy=0x7fc4d01401d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:46.940 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.956+0000 7fc4b7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:16:46.940 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.956+0000 7fc4b7fff640 1 monclient: found mon.a 2024-03-29T11:16:46.940 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.956+0000 7fc4b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:46.940 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.956+0000 7fc4b7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:16:46.940 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.956+0000 7fc4b7fff640 1 -- 172.21.15.47:0/2408031771 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4d01df800 con 0x7fc4d00b4330 2024-03-29T11:16:46.940 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.956+0000 7fc4b7fff640 10 monclient: _finish_auth 0 2024-03-29T11:16:46.940 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.956+0000 7fc4b7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:16:46.940 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.956+0000 7fc4b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:46.940 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.956+0000 7fc4b7fff640 1 -- 172.21.15.47:0/2408031771 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc4c8001c30 con 0x7fc4d00b4330 2024-03-29T11:16:46.940 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.956+0000 7fc4d5f82640 5 monclient: authenticate success, global_id 4359 2024-03-29T11:16:46.940 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.957+0000 7fc4d5f82640 10 monclient: _renew_subs 2024-03-29T11:16:46.940 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.957+0000 7fc4d5f82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:16:46.941 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.957+0000 7fc4d5f82640 1 -- 172.21.15.47:0/2408031771 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4d0141430 con 0x7fc4d00b4330 2024-03-29T11:16:46.941 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.957+0000 7fc4d5f82640 10 client.4359.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:16:46.941 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.957+0000 7fc4d5f82640 10 monclient: _renew_subs 2024-03-29T11:16:46.941 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.957+0000 7fc4d5f82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:16:46.941 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.957+0000 7fc4d5f82640 1 -- 172.21.15.47:0/2408031771 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4d01ce0e0 con 0x7fc4d00b4330 2024-03-29T11:16:46.941 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.957+0000 7fc4d5f82640 1 librados: init done 2024-03-29T11:16:46.941 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.957+0000 7fc4b7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:46.941 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.957+0000 7fc4cdffb640 4 set_mon_vals no callback set 2024-03-29T11:16:46.941 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.959+0000 7fc4b7fff640 1 -- 172.21.15.47:0/2408031771 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fc4c8004be0 con 0x7fc4d00b4330 2024-03-29T11:16:46.941 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.959+0000 7fc4b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:46.941 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.959+0000 7fc4b7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:16:46.941 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.959+0000 7fc4b7fff640 10 monclient: dump: 2024-03-29T11:16:46.941 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:16:46.941 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:16:46.941 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:46.942 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:46.942 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:46.942 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:16:46.942 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:16:46.942 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:16:46.942 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:16:46.942 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:16:46.945 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.960+0000 7fc4b7fff640 1 -- 172.21.15.47:0/2408031771 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7fc4c801ac30 con 0x7fc4d00b4330 2024-03-29T11:16:46.945 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.961+0000 7fc4ceffd640 10 monclient: get_auth_request con 0x7fc4bc06dfc0 auth_method 0 2024-03-29T11:16:46.945 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.961+0000 7fc4b7fff640 1 -- 172.21.15.47:0/2408031771 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7fc4c8084be0 con 0x7fc4d00b4330 2024-03-29T11:16:46.945 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.961+0000 7fc4b7fff640 10 client.4359.objecter ms_dispatch 0x7fc4d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:16:46.945 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.961+0000 7fc4b7fff640 3 client.4359.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:16:46.945 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.961+0000 7fc4b7fff640 3 client.4359.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:16:46.945 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.961+0000 7fc4b7fff640 20 client.4359.objecter dump_active .. 0 homeless 2024-03-29T11:16:46.946 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.961+0000 7fc4b7fff640 10 client.4359.objecter ms_handle_connect 0x7fc4bc06dfc0 2024-03-29T11:16:46.946 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.963+0000 7fc4d5f82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:46.946 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.963+0000 7fc4d5f82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:46.946 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.963+0000 7fc4d5f82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:16:46.946 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.963+0000 7fc4d5f82640 1 -- 172.21.15.47:0/2408031771 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4b0005080 con 0x7fc4d00b4330 2024-03-29T11:16:46.954 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.976+0000 7fc4b7fff640 1 -- 172.21.15.47:0/2408031771 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc4c80502f0 con 0x7fc4d00b4330 2024-03-29T11:16:46.954 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.976+0000 7fc4b7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:46.954 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:46.976+0000 7fc4b7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:16:47.285 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:47.307+0000 7fc4d5f82640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:47.285 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:47.307+0000 7fc4d5f82640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:47.285 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:47.307+0000 7fc4d5f82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:16:47.285 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:47.307+0000 7fc4d5f82640 1 -- 172.21.15.47:0/2408031771 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc4b0005330 con 0x7fc4d00b4330 2024-03-29T11:16:47.288 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:47.310+0000 7fc4b7fff640 1 -- 172.21.15.47:0/2408031771 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7fc4c80523f0 con 0x7fc4d00b4330 2024-03-29T11:16:47.288 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:47.310+0000 7fc4b7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:47.288 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:47.310+0000 7fc4b7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:16:47.288 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:16:47.288 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:16:47.291 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:47.314+0000 7fc4d5f82640 10 librados: watch_flush enter 2024-03-29T11:16:47.291 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:47.314+0000 7fc4d5f82640 10 librados: watch_flush exit 2024-03-29T11:16:47.291 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:47.314+0000 7fc4d5f82640 20 client.4359.objecter shutdown clearing up homeless session... 2024-03-29T11:16:47.291 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:47.314+0000 7fc4d5f82640 10 client.4359.objecter successfully canceled tick 2024-03-29T11:16:47.292 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:47.314+0000 7fc4d5f82640 1 -- 172.21.15.47:0/2408031771 >> v1:172.21.15.121:6816/1964117370 conn(0x7fc4bc06dfc0 legacy=0x7fc4bc070480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:47.292 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:47.314+0000 7fc4d5f82640 10 monclient: shutdown 2024-03-29T11:16:47.292 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:47.314+0000 7fc4d5f82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:47.292 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:47.314+0000 7fc4d5f82640 1 -- 172.21.15.47:0/2408031771 >> v1:172.21.15.47:6789/0 conn(0x7fc4d00b4330 legacy=0x7fc4d01dedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:47.295 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:47.318+0000 7fc4d5f82640 1 -- 172.21.15.47:0/2408031771 shutdown_connections 2024-03-29T11:16:47.295 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:47.318+0000 7fc4d5f82640 1 -- 172.21.15.47:0/2408031771 >> 172.21.15.47:0/2408031771 conn(0x7fc4d0066a00 msgr2=0x7fc4d01d14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:47.295 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:47.318+0000 7fc4d5f82640 1 -- 172.21.15.47:0/2408031771 shutdown_connections 2024-03-29T11:16:47.295 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:47.318+0000 7fc4d5f82640 1 -- 172.21.15.47:0/2408031771 wait complete. 2024-03-29T11:16:47.298 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:47.320+0000 7fc4d5f82640 1 librados: shutdown 2024-03-29T11:16:47.318 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:16:54.125 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:16:54.300 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.322+0000 7fc3ed338640 10 monclient: get_monmap_and_config 2024-03-29T11:16:54.300 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.322+0000 7fc3ed338640 10 monclient: build_initial_monmap 2024-03-29T11:16:54.300 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.322+0000 7fc3ed338640 10 monclient: monmap: 2024-03-29T11:16:54.301 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:16:54.301 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:16:54.301 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:16:54.323267+0000 2024-03-29T11:16:54.301 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:16:54.323267+0000 2024-03-29T11:16:54.301 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:54.301 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:16:54.301 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:16:54.301 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:16:54.301 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:16:54.301 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:16:54.302 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.323+0000 7fc3ed338640 1 Processor -- start 2024-03-29T11:16:54.302 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.323+0000 7fc3ed338640 1 -- start start 2024-03-29T11:16:54.302 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.323+0000 7fc3ed338640 10 monclient: init 2024-03-29T11:16:54.302 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.324+0000 7fc3ed338640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:54.302 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.324+0000 7fc3ed338640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:16:54.310 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.324+0000 7fc3ed338640 10 monclient(hunting): picked mon.noname-a con 0x7fc3e80ae250 addr v1:172.21.15.47:6789/0 2024-03-29T11:16:54.310 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.324+0000 7fc3ed338640 10 monclient(hunting): picked mon.noname-b con 0x7fc3e813f970 addr v1:172.21.15.47:6790/0 2024-03-29T11:16:54.310 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.324+0000 7fc3ed338640 10 monclient(hunting): picked mon.noname-c con 0x7fc3e8142800 addr v1:172.21.15.121:6789/0 2024-03-29T11:16:54.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.324+0000 7fc3ed338640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3e80afe00 con 0x7fc3e80ae250 2024-03-29T11:16:54.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.324+0000 7fc3ed338640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3e8142c10 con 0x7fc3e8142800 2024-03-29T11:16:54.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.324+0000 7fc3ed338640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3e80afb20 con 0x7fc3e813f970 2024-03-29T11:16:54.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.324+0000 7fc3ed338640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:54.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.324+0000 7fc3ed338640 10 monclient(hunting): authenticate will time out at 1057.512796s 2024-03-29T11:16:54.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.328+0000 7fc3e77fe640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fc3e8142800 0x7fc3e814da10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.47:59645/0 (socket says 172.21.15.47:59645) 2024-03-29T11:16:54.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.328+0000 7fc3e77fe640 1 -- 172.21.15.47:0/880453928 learned_addr learned my addr 172.21.15.47:0/880453928 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:16:54.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.330+0000 7fc3e5ffb640 1 -- 172.21.15.47:0/880453928 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 256529402 0 0) 0x7fc3e80afb20 con 0x7fc3e813f970 2024-03-29T11:16:54.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.330+0000 7fc3e5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:16:54.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.330+0000 7fc3e5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:54.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.330+0000 7fc3e5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:54.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.330+0000 7fc3e5ffb640 10 monclient(hunting): my global_id is 4394 2024-03-29T11:16:54.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.330+0000 7fc3e5ffb640 1 -- 172.21.15.47:0/880453928 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3d0003520 con 0x7fc3e813f970 2024-03-29T11:16:54.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.330+0000 7fc3e5ffb640 1 -- 172.21.15.47:0/880453928 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1502800124 0 0) 0x7fc3d0003520 con 0x7fc3e813f970 2024-03-29T11:16:54.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.330+0000 7fc3e5ffb640 1 -- 172.21.15.47:0/880453928 >> v1:172.21.15.121:6789/0 conn(0x7fc3e8142800 legacy=0x7fc3e814da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:54.312 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.330+0000 7fc3e5ffb640 1 -- 172.21.15.47:0/880453928 >> v1:172.21.15.47:6789/0 conn(0x7fc3e80ae250 legacy=0x7fc3e813f260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:16:54.312 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.331+0000 7fc3e5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:16:54.312 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.331+0000 7fc3e5ffb640 1 monclient: found mon.noname-b 2024-03-29T11:16:54.312 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.331+0000 7fc3e5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:54.312 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.331+0000 7fc3e5ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-29T11:16:54.312 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.331+0000 7fc3e5ffb640 1 -- 172.21.15.47:0/880453928 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e80aab80 con 0x7fc3e813f970 2024-03-29T11:16:54.312 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.331+0000 7fc3e5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:16:54.312 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.331+0000 7fc3e5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:16:54.312 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.331+0000 7fc3e5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:54.312 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.331+0000 7fc3ed338640 5 monclient: authenticate success, global_id 4394 2024-03-29T11:16:54.312 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.331+0000 7fc3ed338640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:16:54.313 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.331+0000 7fc3e5ffb640 1 -- 172.21.15.47:0/880453928 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fc3dc001670 con 0x7fc3e813f970 2024-03-29T11:16:54.313 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.331+0000 7fc3e5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:54.313 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.331+0000 7fc3e5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:16:54.313 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.331+0000 7fc3e5ffb640 10 monclient: dump: 2024-03-29T11:16:54.313 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:16:54.313 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:16:54.313 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:54.313 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:54.313 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:54.313 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:16:54.313 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:16:54.313 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:16:54.314 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:16:54.314 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:16:54.314 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.331+0000 7fc3ed338640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:16:54.315 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.334+0000 7fc3e5ffb640 1 -- 172.21.15.47:0/880453928 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc3dc001900 con 0x7fc3e813f970 2024-03-29T11:16:54.315 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.334+0000 7fc3e5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:54.315 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.334+0000 7fc3ed338640 10 monclient: get_monmap_and_config success 2024-03-29T11:16:54.315 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.334+0000 7fc3ed338640 4 set_mon_vals no callback set 2024-03-29T11:16:54.315 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.335+0000 7fc3ed338640 10 monclient: shutdown 2024-03-29T11:16:54.315 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.335+0000 7fc3ed338640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:54.316 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.335+0000 7fc3ed338640 1 -- 172.21.15.47:0/880453928 >> v1:172.21.15.47:6790/0 conn(0x7fc3e813f970 legacy=0x7fc3e81420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:54.316 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.335+0000 7fc3ed338640 1 -- 172.21.15.47:0/880453928 shutdown_connections 2024-03-29T11:16:54.316 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.335+0000 7fc3ed338640 1 -- 172.21.15.47:0/880453928 >> 172.21.15.47:0/880453928 conn(0x7fc3e80b35e0 msgr2=0x7fc3e80b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:54.316 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.336+0000 7fc3ed338640 1 -- 172.21.15.47:0/880453928 shutdown_connections 2024-03-29T11:16:54.316 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.336+0000 7fc3ed338640 1 -- 172.21.15.47:0/880453928 wait complete. 2024-03-29T11:16:54.316 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.336+0000 7fc3ed338640 10 monclient: build_initial_monmap 2024-03-29T11:16:54.316 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.336+0000 7fc3ed338640 10 monclient: monmap: 2024-03-29T11:16:54.316 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:16:54.316 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:16:54.316 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:16:54.316 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:16:54.316 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:54.316 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:16:54.316 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:16:54.316 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:16:54.317 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:16:54.317 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:16:54.317 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.337+0000 7fc3ed338640 1 librados: starting msgr at 2024-03-29T11:16:54.317 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.337+0000 7fc3ed338640 1 librados: starting objecter 2024-03-29T11:16:54.317 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.337+0000 7fc3ed338640 1 Processor -- start 2024-03-29T11:16:54.317 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.337+0000 7fc3ed338640 1 -- start start 2024-03-29T11:16:54.317 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.337+0000 7fc3ed338640 1 librados: setting wanted keys 2024-03-29T11:16:54.317 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.337+0000 7fc3ed338640 1 librados: calling monclient init 2024-03-29T11:16:54.317 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.337+0000 7fc3ed338640 10 monclient: init 2024-03-29T11:16:54.319 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.338+0000 7fc3ed338640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:54.319 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.338+0000 7fc3ed338640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:16:54.319 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.338+0000 7fc3ed338640 10 monclient(hunting): picked mon.noname-a con 0x7fc3e80ae250 addr v1:172.21.15.47:6789/0 2024-03-29T11:16:54.319 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.338+0000 7fc3ed338640 10 monclient(hunting): picked mon.noname-c con 0x7fc3e813f970 addr v1:172.21.15.47:6790/0 2024-03-29T11:16:54.319 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.338+0000 7fc3ed338640 10 monclient(hunting): picked mon.noname-b con 0x7fc3e8142800 addr v1:172.21.15.121:6789/0 2024-03-29T11:16:54.319 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.338+0000 7fc3ed338640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3e813b910 con 0x7fc3e80ae250 2024-03-29T11:16:54.319 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.338+0000 7fc3ed338640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3e8145af0 con 0x7fc3e8142800 2024-03-29T11:16:54.319 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.338+0000 7fc3ed338640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3e8145dd0 con 0x7fc3e813f970 2024-03-29T11:16:54.319 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.338+0000 7fc3ed338640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:54.319 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.338+0000 7fc3ed338640 10 monclient(hunting): authenticate will time out at 1057.526614s 2024-03-29T11:16:54.329 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.343+0000 7fc3e6ffd640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fc3e80ae250 0x7fc3e8143d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:59965/0 (socket says 172.21.15.47:59965) 2024-03-29T11:16:54.329 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.344+0000 7fc3e6ffd640 1 -- 172.21.15.47:0/2981285763 learned_addr learned my addr 172.21.15.47:0/2981285763 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:16:54.329 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.344+0000 7fc3cf7fe640 10 client.?.objecter ms_handle_connect 0x7fc3e80ae250 2024-03-29T11:16:54.329 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.344+0000 7fc3cf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:54.329 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.344+0000 7fc3cf7fe640 1 -- 172.21.15.47:0/2981285763 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 610640605 0 0) 0x7fc3e813b910 con 0x7fc3e80ae250 2024-03-29T11:16:54.329 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.344+0000 7fc3cf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:16:54.329 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.344+0000 7fc3cf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:54.329 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.344+0000 7fc3cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:54.329 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.344+0000 7fc3cf7fe640 10 monclient(hunting): my global_id is 4362 2024-03-29T11:16:54.329 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.344+0000 7fc3cf7fe640 1 -- 172.21.15.47:0/2981285763 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3d4004eb0 con 0x7fc3e80ae250 2024-03-29T11:16:54.329 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.344+0000 7fc3cf7fe640 10 client.?.objecter ms_handle_connect 0x7fc3e813f970 2024-03-29T11:16:54.330 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.344+0000 7fc3cf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:54.330 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.345+0000 7fc3cf7fe640 1 -- 172.21.15.47:0/2981285763 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2196017050 0 0) 0x7fc3d4004eb0 con 0x7fc3e80ae250 2024-03-29T11:16:54.330 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.345+0000 7fc3cf7fe640 1 -- 172.21.15.47:0/2981285763 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3e813b910 con 0x7fc3e80ae250 2024-03-29T11:16:54.330 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.345+0000 7fc3cf7fe640 10 client.?.objecter ms_handle_connect 0x7fc3e8142800 2024-03-29T11:16:54.330 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.345+0000 7fc3cf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:54.330 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.345+0000 7fc3cf7fe640 1 -- 172.21.15.47:0/2981285763 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fc3e0001ce0 con 0x7fc3e80ae250 2024-03-29T11:16:54.330 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.345+0000 7fc3cf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:54.330 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.345+0000 7fc3cf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:16:54.330 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.345+0000 7fc3cf7fe640 10 monclient(hunting): dump: 2024-03-29T11:16:54.330 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:16:54.330 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:16:54.330 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:54.330 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:54.331 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:54.331 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:16:54.331 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:16:54.331 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:16:54.331 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:16:54.331 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:16:54.331 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.345+0000 7fc3cf7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:16:54.331 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.345+0000 7fc3cf7fe640 1 -- 172.21.15.47:0/2981285763 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3242625754 0 0) 0x7fc3e8145dd0 con 0x7fc3e813f970 2024-03-29T11:16:54.331 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.345+0000 7fc3cf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:16:54.331 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.345+0000 7fc3cf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:54.331 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.345+0000 7fc3cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:54.331 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.345+0000 7fc3cf7fe640 10 monclient(hunting): my global_id is 4397 2024-03-29T11:16:54.331 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.345+0000 7fc3cf7fe640 1 -- 172.21.15.47:0/2981285763 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3d4004eb0 con 0x7fc3e813f970 2024-03-29T11:16:54.331 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.347+0000 7fc3cf7fe640 1 -- 172.21.15.47:0/2981285763 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 371627986 0 0) 0x7fc3e8145af0 con 0x7fc3e8142800 2024-03-29T11:16:54.332 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.347+0000 7fc3cf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:16:54.332 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.347+0000 7fc3cf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:54.332 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.347+0000 7fc3cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:54.332 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.347+0000 7fc3cf7fe640 10 monclient(hunting): my global_id is 4378 2024-03-29T11:16:54.332 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.347+0000 7fc3cf7fe640 1 -- 172.21.15.47:0/2981285763 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3e8145dd0 con 0x7fc3e8142800 2024-03-29T11:16:54.332 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.347+0000 7fc3cf7fe640 1 -- 172.21.15.47:0/2981285763 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4293022597 0 0) 0x7fc3d4004eb0 con 0x7fc3e813f970 2024-03-29T11:16:54.332 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.347+0000 7fc3cf7fe640 1 -- 172.21.15.47:0/2981285763 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3e8145af0 con 0x7fc3e813f970 2024-03-29T11:16:54.332 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.347+0000 7fc3cf7fe640 1 -- 172.21.15.47:0/2981285763 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fc3dc003850 con 0x7fc3e813f970 2024-03-29T11:16:54.332 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.347+0000 7fc3cf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:54.332 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.347+0000 7fc3cf7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:16:54.332 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.347+0000 7fc3cf7fe640 10 monclient(hunting): dump: 2024-03-29T11:16:54.333 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:16:54.333 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:16:54.333 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:54.333 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:54.333 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:54.333 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:16:54.333 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:16:54.333 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:16:54.333 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:16:54.333 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:16:54.333 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.347+0000 7fc3cf7fe640 1 -- 172.21.15.47:0/2981285763 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1177365191 0 0) 0x7fc3e813b910 con 0x7fc3e80ae250 2024-03-29T11:16:54.333 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.347+0000 7fc3cf7fe640 1 -- 172.21.15.47:0/2981285763 >> v1:172.21.15.47:6790/0 conn(0x7fc3e813f970 legacy=0x7fc3e8144660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:54.333 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.347+0000 7fc3cf7fe640 1 -- 172.21.15.47:0/2981285763 >> v1:172.21.15.121:6789/0 conn(0x7fc3e8142800 legacy=0x7fc3e81e6650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:54.333 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.347+0000 7fc3cf7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:16:54.334 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.347+0000 7fc3cf7fe640 1 monclient: found mon.a 2024-03-29T11:16:54.334 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.347+0000 7fc3cf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:54.334 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.347+0000 7fc3cf7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:16:54.334 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.347+0000 7fc3cf7fe640 1 -- 172.21.15.47:0/2981285763 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e81460b0 con 0x7fc3e80ae250 2024-03-29T11:16:54.334 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.347+0000 7fc3cf7fe640 10 monclient: _finish_auth 0 2024-03-29T11:16:54.334 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.347+0000 7fc3cf7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:16:54.334 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.348+0000 7fc3cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:54.334 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.348+0000 7fc3ed338640 5 monclient: authenticate success, global_id 4362 2024-03-29T11:16:54.334 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.348+0000 7fc3ed338640 10 monclient: _renew_subs 2024-03-29T11:16:54.334 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.348+0000 7fc3ed338640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:16:54.334 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.348+0000 7fc3ed338640 1 -- 172.21.15.47:0/2981285763 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3e8146390 con 0x7fc3e80ae250 2024-03-29T11:16:54.334 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.348+0000 7fc3ed338640 10 client.4362.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:16:54.334 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.348+0000 7fc3ed338640 10 monclient: _renew_subs 2024-03-29T11:16:54.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.348+0000 7fc3ed338640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:16:54.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.348+0000 7fc3ed338640 1 -- 172.21.15.47:0/2981285763 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3e8145650 con 0x7fc3e80ae250 2024-03-29T11:16:54.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.348+0000 7fc3ed338640 1 librados: init done 2024-03-29T11:16:54.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.349+0000 7fc3cf7fe640 1 -- 172.21.15.47:0/2981285763 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc3e0001fc0 con 0x7fc3e80ae250 2024-03-29T11:16:54.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.349+0000 7fc3cf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:54.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.349+0000 7fc3cf7fe640 1 -- 172.21.15.47:0/2981285763 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fc3e0003dc0 con 0x7fc3e80ae250 2024-03-29T11:16:54.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.349+0000 7fc3cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:54.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.350+0000 7fc3cf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:16:54.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.350+0000 7fc3cf7fe640 10 monclient: dump: 2024-03-29T11:16:54.335 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:16:54.335 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:16:54.335 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:54.335 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:16:54.335 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:54.336 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:16:54.336 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:16:54.336 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:16:54.336 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:16:54.336 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:16:54.336 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.350+0000 7fc3ed338640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:54.336 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.350+0000 7fc3e57fa640 4 set_mon_vals no callback set 2024-03-29T11:16:54.336 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.351+0000 7fc3ed338640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:54.336 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.351+0000 7fc3ed338640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:16:54.336 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.351+0000 7fc3ed338640 1 -- 172.21.15.47:0/2981285763 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3e8075f10 con 0x7fc3e80ae250 2024-03-29T11:16:54.336 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.351+0000 7fc3cf7fe640 1 -- 172.21.15.47:0/2981285763 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7fc3e0003f20 con 0x7fc3e80ae250 2024-03-29T11:16:54.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.352+0000 7fc3cf7fe640 1 -- 172.21.15.47:0/2981285763 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7fc3e0054c50 con 0x7fc3e80ae250 2024-03-29T11:16:54.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.352+0000 7fc3cf7fe640 10 client.4362.objecter ms_dispatch 0x7fc3e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:16:54.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.352+0000 7fc3cf7fe640 3 client.4362.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:16:54.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.352+0000 7fc3cf7fe640 3 client.4362.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:16:54.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.352+0000 7fc3cf7fe640 20 client.4362.objecter dump_active .. 0 homeless 2024-03-29T11:16:54.338 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.352+0000 7fc3e67fc640 10 monclient: get_auth_request con 0x7fc3d406df30 auth_method 0 2024-03-29T11:16:54.338 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.355+0000 7fc3cf7fe640 10 client.4362.objecter ms_handle_connect 0x7fc3d406df30 2024-03-29T11:16:54.340 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.363+0000 7fc3cf7fe640 1 -- 172.21.15.47:0/2981285763 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc3e0050ef0 con 0x7fc3e80ae250 2024-03-29T11:16:54.340 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.363+0000 7fc3cf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:54.341 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.363+0000 7fc3cf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:16:54.664 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.687+0000 7fc3ed338640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:54.664 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.687+0000 7fc3ed338640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:54.665 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.687+0000 7fc3ed338640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:16:54.665 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.687+0000 7fc3ed338640 1 -- 172.21.15.47:0/2981285763 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc3e80b2600 con 0x7fc3e80ae250 2024-03-29T11:16:54.666 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.689+0000 7fc3cf7fe640 1 -- 172.21.15.47:0/2981285763 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7fc3e0051090 con 0x7fc3e80ae250 2024-03-29T11:16:54.666 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.689+0000 7fc3cf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:54.666 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.689+0000 7fc3cf7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:16:54.667 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:16:54.667 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:16:54.670 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.693+0000 7fc3ed338640 10 librados: watch_flush enter 2024-03-29T11:16:54.671 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.694+0000 7fc3ed338640 10 librados: watch_flush exit 2024-03-29T11:16:54.671 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.694+0000 7fc3ed338640 20 client.4362.objecter shutdown clearing up homeless session... 2024-03-29T11:16:54.671 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.694+0000 7fc3ed338640 10 client.4362.objecter successfully canceled tick 2024-03-29T11:16:54.672 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.695+0000 7fc3ed338640 1 -- 172.21.15.47:0/2981285763 >> v1:172.21.15.121:6816/1964117370 conn(0x7fc3d406df30 legacy=0x7fc3d40703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:54.672 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.695+0000 7fc3ed338640 10 monclient: shutdown 2024-03-29T11:16:54.673 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.695+0000 7fc3ed338640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:54.673 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.695+0000 7fc3ed338640 1 -- 172.21.15.47:0/2981285763 >> v1:172.21.15.47:6789/0 conn(0x7fc3e80ae250 legacy=0x7fc3e8143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:54.673 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.695+0000 7fc3ed338640 1 -- 172.21.15.47:0/2981285763 shutdown_connections 2024-03-29T11:16:54.673 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.695+0000 7fc3ed338640 1 -- 172.21.15.47:0/2981285763 >> 172.21.15.47:0/2981285763 conn(0x7fc3e80b35e0 msgr2=0x7fc3e80b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:54.673 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.696+0000 7fc3ed338640 1 -- 172.21.15.47:0/2981285763 shutdown_connections 2024-03-29T11:16:54.673 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.696+0000 7fc3ed338640 1 -- 172.21.15.47:0/2981285763 wait complete. 2024-03-29T11:16:54.674 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:16:54.696+0000 7fc3ed338640 1 librados: shutdown 2024-03-29T11:16:54.692 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:17:01.498 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:17:01.672 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.694+0000 7f0bb40d9640 10 monclient: get_monmap_and_config 2024-03-29T11:17:01.672 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.694+0000 7f0bb40d9640 10 monclient: build_initial_monmap 2024-03-29T11:17:01.672 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.694+0000 7f0bb40d9640 10 monclient: monmap: 2024-03-29T11:17:01.672 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:17:01.672 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:17:01.673 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:17:01.695900+0000 2024-03-29T11:17:01.673 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:17:01.695900+0000 2024-03-29T11:17:01.673 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:01.673 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:17:01.673 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:17:01.673 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:17:01.674 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:17:01.674 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:17:01.675 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.698+0000 7f0bb40d9640 1 Processor -- start 2024-03-29T11:17:01.678 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.699+0000 7f0bb40d9640 1 -- start start 2024-03-29T11:17:01.678 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.699+0000 7f0bb40d9640 10 monclient: init 2024-03-29T11:17:01.679 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.700+0000 7f0bb40d9640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:01.679 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.700+0000 7f0bb40d9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:17:01.679 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.700+0000 7f0bb40d9640 10 monclient(hunting): picked mon.noname-a con 0x7f0bac1457f0 addr v1:172.21.15.47:6789/0 2024-03-29T11:17:01.679 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.700+0000 7f0bb40d9640 10 monclient(hunting): picked mon.noname-b con 0x7f0bac13f650 addr v1:172.21.15.47:6790/0 2024-03-29T11:17:01.679 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.700+0000 7f0bb40d9640 10 monclient(hunting): picked mon.noname-c con 0x7f0bac0a9d70 addr v1:172.21.15.121:6789/0 2024-03-29T11:17:01.679 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.700+0000 7f0bb40d9640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bac0aaef0 con 0x7f0bac1457f0 2024-03-29T11:17:01.680 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.700+0000 7f0bb40d9640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bac145bd0 con 0x7f0bac0a9d70 2024-03-29T11:17:01.680 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.700+0000 7f0bb40d9640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bac145eb0 con 0x7f0bac13f650 2024-03-29T11:17:01.680 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.700+0000 7f0bb40d9640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:01.680 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.700+0000 7f0bb40d9640 10 monclient(hunting): authenticate will time out at 1064.888733s 2024-03-29T11:17:01.683 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.706+0000 7f0bb2676640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f0bac0a9d70 0x7f0bac14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.47:36149/0 (socket says 172.21.15.47:36149) 2024-03-29T11:17:01.683 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.706+0000 7f0bb2676640 1 -- 172.21.15.47:0/3271912303 learned_addr learned my addr 172.21.15.47:0/3271912303 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:17:01.690 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.708+0000 7f0bb0e73640 1 -- 172.21.15.47:0/3271912303 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1285292975 0 0) 0x7f0bac145bd0 con 0x7f0bac0a9d70 2024-03-29T11:17:01.690 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.708+0000 7f0bb0e73640 10 monclient(hunting): _negotiate 2024-03-29T11:17:01.690 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.708+0000 7f0bb0e73640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:01.690 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.708+0000 7f0bb0e73640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:01.690 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.708+0000 7f0bb0e73640 10 monclient(hunting): my global_id is 4381 2024-03-29T11:17:01.691 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.708+0000 7f0bb0e73640 1 -- 172.21.15.47:0/3271912303 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b98003520 con 0x7f0bac0a9d70 2024-03-29T11:17:01.691 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.709+0000 7f0bb0e73640 1 -- 172.21.15.47:0/3271912303 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 914030050 0 0) 0x7f0bac145eb0 con 0x7f0bac13f650 2024-03-29T11:17:01.691 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.709+0000 7f0bb0e73640 10 monclient(hunting): _negotiate 2024-03-29T11:17:01.691 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.709+0000 7f0bb0e73640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:01.691 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.709+0000 7f0bb0e73640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:01.691 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.709+0000 7f0bb0e73640 10 monclient(hunting): my global_id is 4400 2024-03-29T11:17:01.691 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.709+0000 7f0bb0e73640 1 -- 172.21.15.47:0/3271912303 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bac145bd0 con 0x7f0bac13f650 2024-03-29T11:17:01.691 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.709+0000 7f0bb0e73640 1 -- 172.21.15.47:0/3271912303 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3396096330 0 0) 0x7f0b98003520 con 0x7f0bac0a9d70 2024-03-29T11:17:01.691 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.709+0000 7f0bb0e73640 1 -- 172.21.15.47:0/3271912303 >> v1:172.21.15.47:6790/0 conn(0x7f0bac13f650 legacy=0x7f0bac0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:01.691 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.709+0000 7f0bb0e73640 1 -- 172.21.15.47:0/3271912303 >> v1:172.21.15.47:6789/0 conn(0x7f0bac1457f0 legacy=0x7f0bac13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:17:01.691 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.709+0000 7f0bb0e73640 10 monclient: _finish_hunting 0 2024-03-29T11:17:01.691 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.709+0000 7f0bb0e73640 1 monclient: found mon.noname-c 2024-03-29T11:17:01.691 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.709+0000 7f0bb0e73640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:01.692 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.709+0000 7f0bb0e73640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:17:01.692 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.709+0000 7f0bb0e73640 1 -- 172.21.15.47:0/3271912303 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bac146340 con 0x7f0bac0a9d70 2024-03-29T11:17:01.692 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.709+0000 7f0bb0e73640 10 monclient: _finish_auth 0 2024-03-29T11:17:01.692 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.709+0000 7f0bb0e73640 10 monclient: _check_auth_tickets 2024-03-29T11:17:01.692 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.709+0000 7f0bb0e73640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:01.692 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.709+0000 7f0bb0e73640 1 -- 172.21.15.47:0/3271912303 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f0b9c001690 con 0x7f0bac0a9d70 2024-03-29T11:17:01.692 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.709+0000 7f0bb0e73640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:01.692 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.709+0000 7f0bb0e73640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:17:01.692 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.709+0000 7f0bb0e73640 10 monclient: dump: 2024-03-29T11:17:01.692 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:17:01.692 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:17:01.692 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:01.692 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:01.692 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:01.692 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:17:01.693 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:17:01.693 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:17:01.693 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:17:01.693 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:17:01.693 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.709+0000 7f0bb0e73640 1 -- 172.21.15.47:0/3271912303 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b9c0017f0 con 0x7f0bac0a9d70 2024-03-29T11:17:01.693 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.709+0000 7f0bb0e73640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:01.693 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.709+0000 7f0bb0e73640 1 -- 172.21.15.47:0/3271912303 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f0b9c001c00 con 0x7f0bac0a9d70 2024-03-29T11:17:01.693 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.709+0000 7f0bb0e73640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:01.693 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.709+0000 7f0bb0e73640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:17:01.693 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.709+0000 7f0bb0e73640 10 monclient: dump: 2024-03-29T11:17:01.693 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:17:01.693 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:17:01.693 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:01.693 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:01.693 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:01.694 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:17:01.694 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:17:01.694 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:17:01.694 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:17:01.694 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:17:01.694 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.710+0000 7f0bb40d9640 5 monclient: authenticate success, global_id 4381 2024-03-29T11:17:01.694 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.710+0000 7f0bb40d9640 10 monclient: get_monmap_and_config success 2024-03-29T11:17:01.694 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.710+0000 7f0bb40d9640 4 set_mon_vals no callback set 2024-03-29T11:17:01.694 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.711+0000 7f0bb40d9640 10 monclient: shutdown 2024-03-29T11:17:01.694 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.711+0000 7f0bb40d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:01.694 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.711+0000 7f0bb40d9640 1 -- 172.21.15.47:0/3271912303 >> v1:172.21.15.121:6789/0 conn(0x7f0bac0a9d70 legacy=0x7f0bac14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:01.694 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.711+0000 7f0bb40d9640 1 -- 172.21.15.47:0/3271912303 shutdown_connections 2024-03-29T11:17:01.694 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.711+0000 7f0bb40d9640 1 -- 172.21.15.47:0/3271912303 >> 172.21.15.47:0/3271912303 conn(0x7f0bac066a00 msgr2=0x7f0bac066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:01.695 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.714+0000 7f0bb40d9640 1 -- 172.21.15.47:0/3271912303 shutdown_connections 2024-03-29T11:17:01.695 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.716+0000 7f0bb40d9640 1 -- 172.21.15.47:0/3271912303 wait complete. 2024-03-29T11:17:01.696 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.717+0000 7f0bb40d9640 10 monclient: build_initial_monmap 2024-03-29T11:17:01.696 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.717+0000 7f0bb40d9640 10 monclient: monmap: 2024-03-29T11:17:01.696 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:17:01.696 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:17:01.696 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:17:01.696 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:17:01.696 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:01.696 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:17:01.696 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:17:01.696 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:17:01.697 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:17:01.697 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:17:01.697 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.717+0000 7f0bb40d9640 1 librados: starting msgr at 2024-03-29T11:17:01.697 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.717+0000 7f0bb40d9640 1 librados: starting objecter 2024-03-29T11:17:01.697 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.718+0000 7f0bb40d9640 1 Processor -- start 2024-03-29T11:17:01.699 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.718+0000 7f0bb40d9640 1 -- start start 2024-03-29T11:17:01.699 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.718+0000 7f0bb40d9640 1 librados: setting wanted keys 2024-03-29T11:17:01.699 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.718+0000 7f0bb40d9640 1 librados: calling monclient init 2024-03-29T11:17:01.699 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.718+0000 7f0bb40d9640 10 monclient: init 2024-03-29T11:17:01.699 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.718+0000 7f0bb40d9640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:01.699 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.718+0000 7f0bb40d9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:17:01.699 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.718+0000 7f0bb40d9640 10 monclient(hunting): picked mon.noname-b con 0x7f0bac0a9d70 addr v1:172.21.15.121:6789/0 2024-03-29T11:17:01.699 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.718+0000 7f0bb40d9640 10 monclient(hunting): picked mon.noname-a con 0x7f0bac13f650 addr v1:172.21.15.47:6789/0 2024-03-29T11:17:01.699 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.721+0000 7f0bb40d9640 10 monclient(hunting): picked mon.noname-c con 0x7f0bac1457f0 addr v1:172.21.15.47:6790/0 2024-03-29T11:17:01.699 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.721+0000 7f0bb40d9640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bac06eb40 con 0x7f0bac13f650 2024-03-29T11:17:01.700 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.721+0000 7f0bb40d9640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bac06ee20 con 0x7f0bac0a9d70 2024-03-29T11:17:01.700 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.722+0000 7f0bb40d9640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bac142d50 con 0x7f0bac1457f0 2024-03-29T11:17:01.700 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.722+0000 7f0bb40d9640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:01.700 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.722+0000 7f0bb40d9640 10 monclient(hunting): authenticate will time out at 1064.910098s 2024-03-29T11:17:01.702 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.723+0000 7f0bb2676640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f0bac1457f0 0x7f0bac071590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:59079/0 (socket says 172.21.15.47:59079) 2024-03-29T11:17:01.702 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.723+0000 7f0bb2676640 1 -- 172.21.15.47:0/3463828698 learned_addr learned my addr 172.21.15.47:0/3463828698 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:17:01.702 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.723+0000 7f0ba27fc640 10 client.?.objecter ms_handle_connect 0x7f0bac1457f0 2024-03-29T11:17:01.702 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.723+0000 7f0ba27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:01.702 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.723+0000 7f0ba27fc640 1 -- 172.21.15.47:0/3463828698 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4015623443 0 0) 0x7f0bac142d50 con 0x7f0bac1457f0 2024-03-29T11:17:01.702 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.724+0000 7f0ba27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:17:01.702 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.724+0000 7f0ba27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:01.702 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.724+0000 7f0ba27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:01.703 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.724+0000 7f0ba27fc640 10 monclient(hunting): my global_id is 4403 2024-03-29T11:17:01.703 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.724+0000 7f0ba27fc640 1 -- 172.21.15.47:0/3463828698 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b90004eb0 con 0x7f0bac1457f0 2024-03-29T11:17:01.703 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.724+0000 7f0ba27fc640 1 -- 172.21.15.47:0/3463828698 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2939612517 0 0) 0x7f0b90004eb0 con 0x7f0bac1457f0 2024-03-29T11:17:01.703 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.724+0000 7f0ba27fc640 1 -- 172.21.15.47:0/3463828698 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0bac142d50 con 0x7f0bac1457f0 2024-03-29T11:17:01.703 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.724+0000 7f0ba27fc640 1 -- 172.21.15.47:0/3463828698 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f0b9c001bc0 con 0x7f0bac1457f0 2024-03-29T11:17:01.703 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.724+0000 7f0ba27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:01.703 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.724+0000 7f0ba27fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:17:01.703 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.724+0000 7f0ba27fc640 10 monclient(hunting): dump: 2024-03-29T11:17:01.703 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:17:01.703 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:17:01.703 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:01.703 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:01.703 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:01.703 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:17:01.704 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:17:01.704 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:17:01.704 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:17:01.704 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:17:01.704 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.724+0000 7f0ba27fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:17:01.704 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.725+0000 7f0ba27fc640 1 -- 172.21.15.47:0/3463828698 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2672530172 0 0) 0x7f0bac142d50 con 0x7f0bac1457f0 2024-03-29T11:17:01.704 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.725+0000 7f0ba27fc640 1 -- 172.21.15.47:0/3463828698 >> v1:172.21.15.121:6789/0 conn(0x7f0bac0a9d70 legacy=0x7f0bac141940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:01.704 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.725+0000 7f0ba27fc640 1 -- 172.21.15.47:0/3463828698 >> v1:172.21.15.47:6789/0 conn(0x7f0bac13f650 legacy=0x7f0bac142640 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:17:01.706 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.726+0000 7f0ba27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:17:01.707 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.726+0000 7f0ba27fc640 1 monclient: found mon.c 2024-03-29T11:17:01.707 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.726+0000 7f0ba27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:01.707 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.726+0000 7f0ba27fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:17:01.707 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.726+0000 7f0ba27fc640 1 -- 172.21.15.47:0/3463828698 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bac13fda0 con 0x7f0bac1457f0 2024-03-29T11:17:01.707 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.726+0000 7f0ba27fc640 10 monclient: _finish_auth 0 2024-03-29T11:17:01.707 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.726+0000 7f0ba27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:17:01.707 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.726+0000 7f0ba27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:01.707 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.726+0000 7f0bb40d9640 5 monclient: authenticate success, global_id 4403 2024-03-29T11:17:01.707 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.726+0000 7f0bb40d9640 10 monclient: _renew_subs 2024-03-29T11:17:01.707 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.726+0000 7f0bb40d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:17:01.707 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.726+0000 7f0bb40d9640 1 -- 172.21.15.47:0/3463828698 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bac140050 con 0x7f0bac1457f0 2024-03-29T11:17:01.707 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.727+0000 7f0bb40d9640 10 client.4403.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:17:01.707 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.727+0000 7f0bb40d9640 10 monclient: _renew_subs 2024-03-29T11:17:01.707 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.727+0000 7f0bb40d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:17:01.708 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.727+0000 7f0bb40d9640 1 -- 172.21.15.47:0/3463828698 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bac13a3b0 con 0x7f0bac1457f0 2024-03-29T11:17:01.708 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.727+0000 7f0bb40d9640 1 librados: init done 2024-03-29T11:17:01.708 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.730+0000 7f0ba27fc640 1 -- 172.21.15.47:0/3463828698 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b9c004a90 con 0x7f0bac1457f0 2024-03-29T11:17:01.709 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.730+0000 7f0ba27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:01.709 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.730+0000 7f0ba27fc640 1 -- 172.21.15.47:0/3463828698 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f0b9c004ea0 con 0x7f0bac1457f0 2024-03-29T11:17:01.709 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.730+0000 7f0ba27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:01.709 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.730+0000 7f0ba27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:17:01.709 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.730+0000 7f0ba27fc640 10 monclient: dump: 2024-03-29T11:17:01.709 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:17:01.710 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:17:01.710 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:01.710 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:01.710 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:01.710 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:17:01.711 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:17:01.711 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:17:01.711 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:17:01.711 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:17:01.711 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.730+0000 7f0ba37fe640 4 set_mon_vals no callback set 2024-03-29T11:17:01.712 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.732+0000 7f0bb40d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:01.712 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.732+0000 7f0bb40d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:01.712 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.732+0000 7f0bb40d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:17:01.712 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.732+0000 7f0bb40d9640 1 -- 172.21.15.47:0/3463828698 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b94005080 con 0x7f0bac1457f0 2024-03-29T11:17:01.713 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.732+0000 7f0ba27fc640 1 -- 172.21.15.47:0/3463828698 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7f0b9c005000 con 0x7f0bac1457f0 2024-03-29T11:17:01.713 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.733+0000 7f0ba27fc640 1 -- 172.21.15.47:0/3463828698 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7f0b9c084c70 con 0x7f0bac1457f0 2024-03-29T11:17:01.713 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.733+0000 7f0ba27fc640 10 client.4403.objecter ms_dispatch 0x7f0bac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:17:01.713 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.733+0000 7f0ba27fc640 3 client.4403.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:17:01.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.733+0000 7f0ba27fc640 3 client.4403.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:17:01.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.733+0000 7f0ba27fc640 20 client.4403.objecter dump_active .. 0 homeless 2024-03-29T11:17:01.715 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.737+0000 7f0bb1e75640 10 monclient: get_auth_request con 0x7f0b9006dd00 auth_method 0 2024-03-29T11:17:01.715 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.737+0000 7f0ba27fc640 10 client.4403.objecter ms_handle_connect 0x7f0b9006dd00 2024-03-29T11:17:01.721 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.744+0000 7f0ba27fc640 1 -- 172.21.15.47:0/3463828698 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0b9c050550 con 0x7f0bac1457f0 2024-03-29T11:17:01.721 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.744+0000 7f0ba27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:01.721 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:01.744+0000 7f0ba27fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:17:02.052 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:02.075+0000 7f0bb40d9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:02.052 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:02.075+0000 7f0bb40d9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:02.053 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:02.075+0000 7f0bb40d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:17:02.053 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:02.075+0000 7f0bb40d9640 1 -- 172.21.15.47:0/3463828698 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b94005330 con 0x7f0bac1457f0 2024-03-29T11:17:02.056 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:02.079+0000 7f0ba27fc640 1 -- 172.21.15.47:0/3463828698 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7f0b9c052480 con 0x7f0bac1457f0 2024-03-29T11:17:02.056 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:02.079+0000 7f0ba27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:02.056 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:02.079+0000 7f0ba27fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:17:02.056 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:17:02.057 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:17:02.060 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:02.083+0000 7f0bb40d9640 10 librados: watch_flush enter 2024-03-29T11:17:02.063 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:02.085+0000 7f0bb40d9640 10 librados: watch_flush exit 2024-03-29T11:17:02.063 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:02.085+0000 7f0bb40d9640 20 client.4403.objecter shutdown clearing up homeless session... 2024-03-29T11:17:02.063 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:02.085+0000 7f0bb40d9640 10 client.4403.objecter successfully canceled tick 2024-03-29T11:17:02.064 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:02.086+0000 7f0bb40d9640 1 -- 172.21.15.47:0/3463828698 >> v1:172.21.15.121:6816/1964117370 conn(0x7f0b9006dd00 legacy=0x7f0b900701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:02.064 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:02.086+0000 7f0bb40d9640 10 monclient: shutdown 2024-03-29T11:17:02.064 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:02.086+0000 7f0bb40d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:02.064 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:02.086+0000 7f0bb40d9640 1 -- 172.21.15.47:0/3463828698 >> v1:172.21.15.47:6790/0 conn(0x7f0bac1457f0 legacy=0x7f0bac071590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:02.065 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:02.087+0000 7f0bb40d9640 1 -- 172.21.15.47:0/3463828698 shutdown_connections 2024-03-29T11:17:02.065 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:02.087+0000 7f0bb40d9640 1 -- 172.21.15.47:0/3463828698 >> 172.21.15.47:0/3463828698 conn(0x7f0bac066a00 msgr2=0x7f0bac13e460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:02.066 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:02.087+0000 7f0bb40d9640 1 -- 172.21.15.47:0/3463828698 shutdown_connections 2024-03-29T11:17:02.066 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:02.087+0000 7f0bb40d9640 1 -- 172.21.15.47:0/3463828698 wait complete. 2024-03-29T11:17:02.066 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:02.087+0000 7f0bb40d9640 1 librados: shutdown 2024-03-29T11:17:02.083 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:17:08.889 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:17:09.030 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.052+0000 7fcaeb87b640 10 monclient: get_monmap_and_config 2024-03-29T11:17:09.030 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.052+0000 7fcaeb87b640 10 monclient: build_initial_monmap 2024-03-29T11:17:09.030 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.052+0000 7fcaeb87b640 10 monclient: monmap: 2024-03-29T11:17:09.030 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:17:09.030 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:17:09.030 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:17:09.053779+0000 2024-03-29T11:17:09.030 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:17:09.053779+0000 2024-03-29T11:17:09.030 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:09.031 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:17:09.031 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:17:09.031 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:17:09.031 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:17:09.031 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:17:09.032 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.053+0000 7fcaeb87b640 1 Processor -- start 2024-03-29T11:17:09.032 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.053+0000 7fcaeb87b640 1 -- start start 2024-03-29T11:17:09.032 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.053+0000 7fcaeb87b640 10 monclient: init 2024-03-29T11:17:09.032 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.054+0000 7fcaeb87b640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:09.032 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.054+0000 7fcaeb87b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:17:09.032 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.054+0000 7fcaeb87b640 10 monclient(hunting): picked mon.noname-b con 0x7fcae41457f0 addr v1:172.21.15.47:6790/0 2024-03-29T11:17:09.032 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.054+0000 7fcaeb87b640 10 monclient(hunting): picked mon.noname-a con 0x7fcae413f650 addr v1:172.21.15.47:6789/0 2024-03-29T11:17:09.032 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.054+0000 7fcaeb87b640 10 monclient(hunting): picked mon.noname-c con 0x7fcae40a9d70 addr v1:172.21.15.121:6789/0 2024-03-29T11:17:09.033 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.054+0000 7fcaeb87b640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcae40aaef0 con 0x7fcae413f650 2024-03-29T11:17:09.033 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.054+0000 7fcaeb87b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcae4145bd0 con 0x7fcae40a9d70 2024-03-29T11:17:09.033 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.054+0000 7fcaeb87b640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcae4145eb0 con 0x7fcae41457f0 2024-03-29T11:17:09.033 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.054+0000 7fcaeb87b640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:09.033 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.054+0000 7fcaeb87b640 10 monclient(hunting): authenticate will time out at 1072.242874s 2024-03-29T11:17:09.033 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.055+0000 7fcae8e16640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fcae413f650 0x7fcae40b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:33420/0 (socket says 172.21.15.47:33420) 2024-03-29T11:17:09.033 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.055+0000 7fcae8e16640 1 -- 172.21.15.47:0/3310359409 learned_addr learned my addr 172.21.15.47:0/3310359409 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:17:09.035 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.055+0000 7fcadbfff640 1 -- 172.21.15.47:0/3310359409 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2336405079 0 0) 0x7fcae40aaef0 con 0x7fcae413f650 2024-03-29T11:17:09.035 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.055+0000 7fcadbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:17:09.035 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.055+0000 7fcadbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:09.035 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.055+0000 7fcadbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:09.035 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.055+0000 7fcadbfff640 10 monclient(hunting): my global_id is 4365 2024-03-29T11:17:09.035 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.055+0000 7fcadbfff640 1 -- 172.21.15.47:0/3310359409 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcad0003520 con 0x7fcae413f650 2024-03-29T11:17:09.035 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.056+0000 7fcadbfff640 1 -- 172.21.15.47:0/3310359409 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2258847343 0 0) 0x7fcad0003520 con 0x7fcae413f650 2024-03-29T11:17:09.035 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.056+0000 7fcadbfff640 1 -- 172.21.15.47:0/3310359409 >> v1:172.21.15.47:6790/0 conn(0x7fcae41457f0 legacy=0x7fcae413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:09.035 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.056+0000 7fcadbfff640 1 -- 172.21.15.47:0/3310359409 >> v1:172.21.15.121:6789/0 conn(0x7fcae40a9d70 legacy=0x7fcae414d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:17:09.036 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.057+0000 7fcadbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:17:09.036 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.057+0000 7fcadbfff640 1 monclient: found mon.noname-a 2024-03-29T11:17:09.036 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.057+0000 7fcadbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:09.036 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.057+0000 7fcadbfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-29T11:17:09.036 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.057+0000 7fcadbfff640 1 -- 172.21.15.47:0/3310359409 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcae4146340 con 0x7fcae413f650 2024-03-29T11:17:09.036 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.057+0000 7fcadbfff640 10 monclient: _finish_auth 0 2024-03-29T11:17:09.036 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.057+0000 7fcadbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:17:09.036 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.057+0000 7fcadbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:09.036 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.057+0000 7fcadbfff640 1 -- 172.21.15.47:0/3310359409 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fcae0001670 con 0x7fcae413f650 2024-03-29T11:17:09.036 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.057+0000 7fcadbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:09.036 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.057+0000 7fcadbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:17:09.036 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.058+0000 7fcadbfff640 10 monclient: dump: 2024-03-29T11:17:09.036 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:17:09.036 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:17:09.037 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:09.037 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:09.037 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:09.037 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:17:09.037 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:17:09.037 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:17:09.037 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:17:09.037 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:17:09.037 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.058+0000 7fcaeb87b640 5 monclient: authenticate success, global_id 4365 2024-03-29T11:17:09.037 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.058+0000 7fcaeb87b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:17:09.038 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.059+0000 7fcadbfff640 1 -- 172.21.15.47:0/3310359409 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcae0001840 con 0x7fcae413f650 2024-03-29T11:17:09.038 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.059+0000 7fcadbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:09.038 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.059+0000 7fcadbfff640 1 -- 172.21.15.47:0/3310359409 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fcae0001c50 con 0x7fcae413f650 2024-03-29T11:17:09.038 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.059+0000 7fcadbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:09.039 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.059+0000 7fcadbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:17:09.039 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.059+0000 7fcadbfff640 10 monclient: dump: 2024-03-29T11:17:09.039 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:17:09.039 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:17:09.039 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:09.039 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:09.039 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:09.039 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:17:09.039 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:17:09.039 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:17:09.039 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:17:09.039 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:17:09.039 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.059+0000 7fcaeb87b640 10 monclient: get_monmap_and_config success 2024-03-29T11:17:09.039 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.059+0000 7fcaeb87b640 4 set_mon_vals no callback set 2024-03-29T11:17:09.040 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.060+0000 7fcaeb87b640 10 monclient: shutdown 2024-03-29T11:17:09.040 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.060+0000 7fcaeb87b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:09.040 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.060+0000 7fcaeb87b640 1 -- 172.21.15.47:0/3310359409 >> v1:172.21.15.47:6789/0 conn(0x7fcae413f650 legacy=0x7fcae40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:09.040 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.060+0000 7fcaeb87b640 1 -- 172.21.15.47:0/3310359409 shutdown_connections 2024-03-29T11:17:09.040 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.060+0000 7fcaeb87b640 1 -- 172.21.15.47:0/3310359409 >> 172.21.15.47:0/3310359409 conn(0x7fcae4066a00 msgr2=0x7fcae4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:09.040 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.060+0000 7fcaeb87b640 1 -- 172.21.15.47:0/3310359409 shutdown_connections 2024-03-29T11:17:09.040 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.060+0000 7fcaeb87b640 1 -- 172.21.15.47:0/3310359409 wait complete. 2024-03-29T11:17:09.040 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.061+0000 7fcaeb87b640 10 monclient: build_initial_monmap 2024-03-29T11:17:09.040 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.061+0000 7fcaeb87b640 10 monclient: monmap: 2024-03-29T11:17:09.040 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:17:09.040 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:17:09.040 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:17:09.040 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:17:09.041 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:09.041 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:17:09.041 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:17:09.041 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:17:09.041 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:17:09.041 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:17:09.044 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.063+0000 7fcaeb87b640 1 librados: starting msgr at 2024-03-29T11:17:09.044 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.063+0000 7fcaeb87b640 1 librados: starting objecter 2024-03-29T11:17:09.044 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.063+0000 7fcaeb87b640 1 Processor -- start 2024-03-29T11:17:09.044 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.063+0000 7fcaeb87b640 1 -- start start 2024-03-29T11:17:09.044 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.063+0000 7fcaeb87b640 1 librados: setting wanted keys 2024-03-29T11:17:09.044 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.063+0000 7fcaeb87b640 1 librados: calling monclient init 2024-03-29T11:17:09.044 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.063+0000 7fcaeb87b640 10 monclient: init 2024-03-29T11:17:09.044 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.064+0000 7fcaeb87b640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:09.044 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.064+0000 7fcaeb87b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:17:09.044 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.064+0000 7fcaeb87b640 10 monclient(hunting): picked mon.noname-b con 0x7fcae40a9d70 addr v1:172.21.15.121:6789/0 2024-03-29T11:17:09.044 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.064+0000 7fcaeb87b640 10 monclient(hunting): picked mon.noname-c con 0x7fcae413f650 addr v1:172.21.15.47:6790/0 2024-03-29T11:17:09.044 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.064+0000 7fcaeb87b640 10 monclient(hunting): picked mon.noname-a con 0x7fcae41457f0 addr v1:172.21.15.47:6789/0 2024-03-29T11:17:09.045 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.064+0000 7fcaeb87b640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcae41dabf0 con 0x7fcae41457f0 2024-03-29T11:17:09.045 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.064+0000 7fcae9e18640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fcae41457f0 0x7fcae41e2f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:33742/0 (socket says 172.21.15.47:33742) 2024-03-29T11:17:09.045 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.064+0000 7fcae9e18640 1 -- 172.21.15.47:0/1387597453 learned_addr learned my addr 172.21.15.47:0/1387597453 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:17:09.045 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.064+0000 7fcaeb87b640 1 -- 172.21.15.47:0/1387597453 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcae41403e0 con 0x7fcae40a9d70 2024-03-29T11:17:09.045 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.064+0000 7fcaeb87b640 1 -- 172.21.15.47:0/1387597453 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcae413fda0 con 0x7fcae413f650 2024-03-29T11:17:09.045 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.064+0000 7fcaeb87b640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:09.045 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.064+0000 7fcaeb87b640 10 monclient(hunting): authenticate will time out at 1072.252672s 2024-03-29T11:17:09.045 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.064+0000 7fcad9ffb640 10 client.?.objecter ms_handle_connect 0x7fcae41457f0 2024-03-29T11:17:09.045 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.064+0000 7fcad9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:09.045 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.064+0000 7fcad9ffb640 1 -- 172.21.15.47:0/1387597453 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1456823289 0 0) 0x7fcae41dabf0 con 0x7fcae41457f0 2024-03-29T11:17:09.045 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.064+0000 7fcad9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:17:09.045 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.065+0000 7fcad9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:09.045 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.065+0000 7fcad9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:09.045 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.065+0000 7fcad9ffb640 10 monclient(hunting): my global_id is 4368 2024-03-29T11:17:09.046 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.065+0000 7fcad9ffb640 1 -- 172.21.15.47:0/1387597453 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcac8004eb0 con 0x7fcae41457f0 2024-03-29T11:17:09.046 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.065+0000 7fcad9ffb640 10 client.?.objecter ms_handle_connect 0x7fcae413f650 2024-03-29T11:17:09.046 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.065+0000 7fcad9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:09.046 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.065+0000 7fcad9ffb640 1 -- 172.21.15.47:0/1387597453 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1376013791 0 0) 0x7fcae413fda0 con 0x7fcae413f650 2024-03-29T11:17:09.046 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.065+0000 7fcad9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:17:09.046 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.065+0000 7fcad9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:09.046 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.065+0000 7fcad9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:09.046 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.065+0000 7fcad9ffb640 10 monclient(hunting): my global_id is 4406 2024-03-29T11:17:09.046 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.065+0000 7fcad9ffb640 1 -- 172.21.15.47:0/1387597453 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcae41dabf0 con 0x7fcae413f650 2024-03-29T11:17:09.046 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.065+0000 7fcad9ffb640 1 -- 172.21.15.47:0/1387597453 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1180894261 0 0) 0x7fcac8004eb0 con 0x7fcae41457f0 2024-03-29T11:17:09.046 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.065+0000 7fcad9ffb640 1 -- 172.21.15.47:0/1387597453 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcac8005490 con 0x7fcae41457f0 2024-03-29T11:17:09.046 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.065+0000 7fcad9ffb640 1 -- 172.21.15.47:0/1387597453 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fcadc001b70 con 0x7fcae41457f0 2024-03-29T11:17:09.046 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.065+0000 7fcad9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:09.047 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.065+0000 7fcad9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:17:09.047 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.065+0000 7fcad9ffb640 10 monclient(hunting): dump: 2024-03-29T11:17:09.047 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:17:09.047 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:17:09.047 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:09.047 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:09.047 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:09.047 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:17:09.047 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:17:09.047 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:17:09.047 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:17:09.047 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:17:09.047 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.065+0000 7fcad9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:17:09.047 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.065+0000 7fcad9ffb640 1 -- 172.21.15.47:0/1387597453 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 275077350 0 0) 0x7fcac8005490 con 0x7fcae41457f0 2024-03-29T11:17:09.048 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.065+0000 7fcad9ffb640 1 -- 172.21.15.47:0/1387597453 >> v1:172.21.15.47:6790/0 conn(0x7fcae413f650 legacy=0x7fcae4140940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:09.048 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.065+0000 7fcad9ffb640 1 -- 172.21.15.47:0/1387597453 >> v1:172.21.15.121:6789/0 conn(0x7fcae40a9d70 legacy=0x7fcae41db0b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:17:09.051 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.068+0000 7fcad9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:17:09.051 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.068+0000 7fcad9ffb640 1 monclient: found mon.a 2024-03-29T11:17:09.051 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.068+0000 7fcad9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:09.051 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.068+0000 7fcad9ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:17:09.051 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.068+0000 7fcad9ffb640 1 -- 172.21.15.47:0/1387597453 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcae413ff30 con 0x7fcae41457f0 2024-03-29T11:17:09.051 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.068+0000 7fcad9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:17:09.051 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.068+0000 7fcad9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:17:09.051 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.068+0000 7fcad9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:09.051 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.069+0000 7fcaeb87b640 5 monclient: authenticate success, global_id 4368 2024-03-29T11:17:09.051 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.069+0000 7fcaeb87b640 10 monclient: _renew_subs 2024-03-29T11:17:09.052 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.069+0000 7fcaeb87b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:17:09.052 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.069+0000 7fcaeb87b640 1 -- 172.21.15.47:0/1387597453 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcae4140120 con 0x7fcae41457f0 2024-03-29T11:17:09.052 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.069+0000 7fcaeb87b640 10 client.4368.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:17:09.052 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.069+0000 7fcaeb87b640 10 monclient: _renew_subs 2024-03-29T11:17:09.052 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.069+0000 7fcaeb87b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:17:09.052 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.069+0000 7fcaeb87b640 1 -- 172.21.15.47:0/1387597453 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcae41dbab0 con 0x7fcae41457f0 2024-03-29T11:17:09.052 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.069+0000 7fcaeb87b640 1 librados: init done 2024-03-29T11:17:09.052 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.070+0000 7fcad9ffb640 1 -- 172.21.15.47:0/1387597453 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcadc002080 con 0x7fcae41457f0 2024-03-29T11:17:09.052 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.070+0000 7fcad9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:09.052 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.070+0000 7fcad9ffb640 1 -- 172.21.15.47:0/1387597453 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fcadc004e20 con 0x7fcae41457f0 2024-03-29T11:17:09.052 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.070+0000 7fcad9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:09.052 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.070+0000 7fcad9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:17:09.052 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.070+0000 7fcad9ffb640 10 monclient: dump: 2024-03-29T11:17:09.052 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:17:09.052 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:17:09.053 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:09.053 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:09.053 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:09.053 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:17:09.053 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:17:09.053 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:17:09.053 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:17:09.053 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:17:09.053 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.070+0000 7fcadb7fe640 4 set_mon_vals no callback set 2024-03-29T11:17:09.053 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.071+0000 7fcaeb87b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:09.053 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.071+0000 7fcad9ffb640 1 -- 172.21.15.47:0/1387597453 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7fcadc01adf0 con 0x7fcae41457f0 2024-03-29T11:17:09.053 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.071+0000 7fcaeb87b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:09.053 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.071+0000 7fcaeb87b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:17:09.053 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.071+0000 7fcaeb87b640 1 -- 172.21.15.47:0/1387597453 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcae4075f10 con 0x7fcae41457f0 2024-03-29T11:17:09.054 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.071+0000 7fcad9ffb640 1 -- 172.21.15.47:0/1387597453 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7fcadc084db0 con 0x7fcae41457f0 2024-03-29T11:17:09.054 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.071+0000 7fcad9ffb640 10 client.4368.objecter ms_dispatch 0x7fcae4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:17:09.054 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.071+0000 7fcad9ffb640 3 client.4368.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:17:09.054 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.071+0000 7fcad9ffb640 3 client.4368.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:17:09.054 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.071+0000 7fcad9ffb640 20 client.4368.objecter dump_active .. 0 homeless 2024-03-29T11:17:09.054 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.072+0000 7fcae9617640 10 monclient: get_auth_request con 0x7fcac806df10 auth_method 0 2024-03-29T11:17:09.054 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.073+0000 7fcad9ffb640 10 client.4368.objecter ms_handle_connect 0x7fcac806df10 2024-03-29T11:17:09.060 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.083+0000 7fcad9ffb640 1 -- 172.21.15.47:0/1387597453 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcadc050690 con 0x7fcae41457f0 2024-03-29T11:17:09.060 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.083+0000 7fcad9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:09.060 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.083+0000 7fcad9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:17:09.419 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.441+0000 7fcaeb87b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:09.419 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.441+0000 7fcaeb87b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:09.419 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.441+0000 7fcaeb87b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:17:09.419 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.441+0000 7fcaeb87b640 1 -- 172.21.15.47:0/1387597453 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcae40a9860 con 0x7fcae41457f0 2024-03-29T11:17:09.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.443+0000 7fcad9ffb640 1 -- 172.21.15.47:0/1387597453 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7fcadc0525c0 con 0x7fcae41457f0 2024-03-29T11:17:09.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.443+0000 7fcad9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:09.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.443+0000 7fcad9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:17:09.420 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:17:09.420 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:17:09.425 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.447+0000 7fcaeb87b640 10 librados: watch_flush enter 2024-03-29T11:17:09.425 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.447+0000 7fcaeb87b640 10 librados: watch_flush exit 2024-03-29T11:17:09.425 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.447+0000 7fcaeb87b640 20 client.4368.objecter shutdown clearing up homeless session... 2024-03-29T11:17:09.425 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.447+0000 7fcaeb87b640 10 client.4368.objecter successfully canceled tick 2024-03-29T11:17:09.425 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.447+0000 7fcaeb87b640 1 -- 172.21.15.47:0/1387597453 >> v1:172.21.15.121:6816/1964117370 conn(0x7fcac806df10 legacy=0x7fcac80703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:09.425 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.447+0000 7fcaeb87b640 10 monclient: shutdown 2024-03-29T11:17:09.425 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.448+0000 7fcaeb87b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:09.425 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.448+0000 7fcaeb87b640 1 -- 172.21.15.47:0/1387597453 >> v1:172.21.15.47:6789/0 conn(0x7fcae41457f0 legacy=0x7fcae41e2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:09.425 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.448+0000 7fcaeb87b640 1 -- 172.21.15.47:0/1387597453 shutdown_connections 2024-03-29T11:17:09.425 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.448+0000 7fcaeb87b640 1 -- 172.21.15.47:0/1387597453 >> 172.21.15.47:0/1387597453 conn(0x7fcae4066a00 msgr2=0x7fcae4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:09.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.450+0000 7fcaeb87b640 1 -- 172.21.15.47:0/1387597453 shutdown_connections 2024-03-29T11:17:09.428 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.451+0000 7fcaeb87b640 1 -- 172.21.15.47:0/1387597453 wait complete. 2024-03-29T11:17:09.428 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:09.451+0000 7fcaeb87b640 1 librados: shutdown 2024-03-29T11:17:09.445 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:17:16.252 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:17:16.405 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.428+0000 7fb95142a640 10 monclient: get_monmap_and_config 2024-03-29T11:17:16.405 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.428+0000 7fb95142a640 10 monclient: build_initial_monmap 2024-03-29T11:17:16.406 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.428+0000 7fb95142a640 10 monclient: monmap: 2024-03-29T11:17:16.406 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:17:16.406 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:17:16.406 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:17:16.429233+0000 2024-03-29T11:17:16.406 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:17:16.429233+0000 2024-03-29T11:17:16.406 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:16.406 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:17:16.406 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:17:16.406 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:17:16.406 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:17:16.406 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:17:16.407 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.429+0000 7fb95142a640 1 Processor -- start 2024-03-29T11:17:16.407 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.429+0000 7fb95142a640 1 -- start start 2024-03-29T11:17:16.407 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.429+0000 7fb95142a640 10 monclient: init 2024-03-29T11:17:16.407 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.430+0000 7fb95142a640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:16.407 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.430+0000 7fb95142a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:17:16.407 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.430+0000 7fb95142a640 10 monclient(hunting): picked mon.noname-a con 0x7fb94c0b0770 addr v1:172.21.15.47:6789/0 2024-03-29T11:17:16.407 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.430+0000 7fb95142a640 10 monclient(hunting): picked mon.noname-b con 0x7fb94c14a700 addr v1:172.21.15.47:6790/0 2024-03-29T11:17:16.408 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.430+0000 7fb95142a640 10 monclient(hunting): picked mon.noname-c con 0x7fb94c0ac5a0 addr v1:172.21.15.121:6789/0 2024-03-29T11:17:16.408 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.430+0000 7fb95142a640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb94c0aba70 con 0x7fb94c0b0770 2024-03-29T11:17:16.408 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.430+0000 7fb95142a640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb94c0aecb0 con 0x7fb94c0ac5a0 2024-03-29T11:17:16.408 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.430+0000 7fb95142a640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb94c0af150 con 0x7fb94c14a700 2024-03-29T11:17:16.408 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.430+0000 7fb95142a640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:16.408 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.430+0000 7fb95142a640 10 monclient(hunting): authenticate will time out at 1079.618399s 2024-03-29T11:17:16.408 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.430+0000 7fb94affd640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fb94c0b0770 0x7fb94c149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:55124/0 (socket says 172.21.15.47:55124) 2024-03-29T11:17:16.408 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.430+0000 7fb94affd640 1 -- 172.21.15.47:0/1149764133 learned_addr learned my addr 172.21.15.47:0/1149764133 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:17:16.409 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.431+0000 7fb949ffb640 1 -- 172.21.15.47:0/1149764133 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1232648177 0 0) 0x7fb94c0aecb0 con 0x7fb94c0ac5a0 2024-03-29T11:17:16.409 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.431+0000 7fb949ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:17:16.410 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.431+0000 7fb949ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:16.410 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.431+0000 7fb949ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:16.410 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.431+0000 7fb949ffb640 10 monclient(hunting): my global_id is 4384 2024-03-29T11:17:16.410 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.431+0000 7fb949ffb640 1 -- 172.21.15.47:0/1149764133 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb934003520 con 0x7fb94c0ac5a0 2024-03-29T11:17:16.410 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.431+0000 7fb949ffb640 1 -- 172.21.15.47:0/1149764133 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1732258467 0 0) 0x7fb94c0aba70 con 0x7fb94c0b0770 2024-03-29T11:17:16.410 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.431+0000 7fb949ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:17:16.410 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.431+0000 7fb949ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:16.410 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.431+0000 7fb949ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:16.410 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.431+0000 7fb949ffb640 10 monclient(hunting): my global_id is 4371 2024-03-29T11:17:16.410 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.431+0000 7fb949ffb640 1 -- 172.21.15.47:0/1149764133 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb94c0aecb0 con 0x7fb94c0b0770 2024-03-29T11:17:16.410 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.431+0000 7fb949ffb640 1 -- 172.21.15.47:0/1149764133 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1782068999 0 0) 0x7fb934003520 con 0x7fb94c0ac5a0 2024-03-29T11:17:16.410 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.431+0000 7fb949ffb640 1 -- 172.21.15.47:0/1149764133 >> v1:172.21.15.47:6790/0 conn(0x7fb94c14a700 legacy=0x7fb94c0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:16.411 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.431+0000 7fb949ffb640 1 -- 172.21.15.47:0/1149764133 >> v1:172.21.15.47:6789/0 conn(0x7fb94c0b0770 legacy=0x7fb94c149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:16.411 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.431+0000 7fb949ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:17:16.411 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.431+0000 7fb949ffb640 1 monclient: found mon.noname-c 2024-03-29T11:17:16.411 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.431+0000 7fb949ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:16.411 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.432+0000 7fb949ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:17:16.411 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.432+0000 7fb949ffb640 1 -- 172.21.15.47:0/1149764133 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb94c0ac9d0 con 0x7fb94c0ac5a0 2024-03-29T11:17:16.411 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.432+0000 7fb949ffb640 10 monclient: _finish_auth 0 2024-03-29T11:17:16.411 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.432+0000 7fb949ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:17:16.411 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.432+0000 7fb949ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:16.411 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.432+0000 7fb949ffb640 1 -- 172.21.15.47:0/1149764133 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fb944001a30 con 0x7fb94c0ac5a0 2024-03-29T11:17:16.411 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.432+0000 7fb949ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:16.411 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.432+0000 7fb949ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:17:16.411 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.432+0000 7fb949ffb640 10 monclient: dump: 2024-03-29T11:17:16.411 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:17:16.412 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:17:16.412 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:16.412 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:16.412 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:16.412 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:17:16.412 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:17:16.412 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:17:16.412 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:17:16.412 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:17:16.412 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.432+0000 7fb95142a640 5 monclient: authenticate success, global_id 4384 2024-03-29T11:17:16.412 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.432+0000 7fb95142a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:17:16.412 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.432+0000 7fb949ffb640 1 -- 172.21.15.47:0/1149764133 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb944001b90 con 0x7fb94c0ac5a0 2024-03-29T11:17:16.412 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.432+0000 7fb949ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:16.412 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.432+0000 7fb949ffb640 1 -- 172.21.15.47:0/1149764133 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fb944001f60 con 0x7fb94c0ac5a0 2024-03-29T11:17:16.413 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.432+0000 7fb949ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:16.413 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.432+0000 7fb949ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:17:16.413 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.432+0000 7fb949ffb640 10 monclient: dump: 2024-03-29T11:17:16.413 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:17:16.413 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:17:16.413 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:16.413 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:16.413 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:16.413 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:17:16.413 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:17:16.413 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:17:16.413 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:17:16.413 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:17:16.413 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.432+0000 7fb95142a640 10 monclient: get_monmap_and_config success 2024-03-29T11:17:16.413 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.432+0000 7fb95142a640 4 set_mon_vals no callback set 2024-03-29T11:17:16.417 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.433+0000 7fb95142a640 10 monclient: shutdown 2024-03-29T11:17:16.417 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.433+0000 7fb95142a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:16.417 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.433+0000 7fb95142a640 1 -- 172.21.15.47:0/1149764133 >> v1:172.21.15.121:6789/0 conn(0x7fb94c0ac5a0 legacy=0x7fb94c151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:16.417 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.433+0000 7fb95142a640 1 -- 172.21.15.47:0/1149764133 shutdown_connections 2024-03-29T11:17:16.417 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.433+0000 7fb95142a640 1 -- 172.21.15.47:0/1149764133 >> 172.21.15.47:0/1149764133 conn(0x7fb94c06b660 msgr2=0x7fb94c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:16.417 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.433+0000 7fb95142a640 1 -- 172.21.15.47:0/1149764133 shutdown_connections 2024-03-29T11:17:16.417 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.433+0000 7fb95142a640 1 -- 172.21.15.47:0/1149764133 wait complete. 2024-03-29T11:17:16.418 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.434+0000 7fb95142a640 10 monclient: build_initial_monmap 2024-03-29T11:17:16.418 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.434+0000 7fb95142a640 10 monclient: monmap: 2024-03-29T11:17:16.418 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:17:16.418 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:17:16.418 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:17:16.418 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:17:16.418 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:16.418 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:17:16.418 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:17:16.418 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:17:16.418 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:17:16.418 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:17:16.418 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.434+0000 7fb95142a640 1 librados: starting msgr at 2024-03-29T11:17:16.418 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.434+0000 7fb95142a640 1 librados: starting objecter 2024-03-29T11:17:16.419 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.435+0000 7fb95142a640 1 Processor -- start 2024-03-29T11:17:16.419 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.435+0000 7fb95142a640 1 -- start start 2024-03-29T11:17:16.419 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.435+0000 7fb95142a640 1 librados: setting wanted keys 2024-03-29T11:17:16.419 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.435+0000 7fb95142a640 1 librados: calling monclient init 2024-03-29T11:17:16.419 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.435+0000 7fb95142a640 10 monclient: init 2024-03-29T11:17:16.419 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.435+0000 7fb95142a640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:16.419 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.435+0000 7fb95142a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:17:16.419 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.435+0000 7fb95142a640 10 monclient(hunting): picked mon.noname-b con 0x7fb94c0ac5a0 addr v1:172.21.15.121:6789/0 2024-03-29T11:17:16.419 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.435+0000 7fb95142a640 10 monclient(hunting): picked mon.noname-a con 0x7fb94c0b0770 addr v1:172.21.15.47:6789/0 2024-03-29T11:17:16.419 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.435+0000 7fb95142a640 10 monclient(hunting): picked mon.noname-c con 0x7fb94c14a700 addr v1:172.21.15.47:6790/0 2024-03-29T11:17:16.419 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.435+0000 7fb95142a640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb94c126570 con 0x7fb94c0b0770 2024-03-29T11:17:16.419 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.435+0000 7fb95142a640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb94c145bb0 con 0x7fb94c0ac5a0 2024-03-29T11:17:16.419 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.435+0000 7fb95142a640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb94c1286b0 con 0x7fb94c14a700 2024-03-29T11:17:16.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.435+0000 7fb95142a640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:16.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.435+0000 7fb95142a640 10 monclient(hunting): authenticate will time out at 1079.623864s 2024-03-29T11:17:16.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.436+0000 7fb94a7fc640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fb94c0b0770 0x7fb94c144d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:55272/0 (socket says 172.21.15.47:55272) 2024-03-29T11:17:16.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.436+0000 7fb94a7fc640 1 -- 172.21.15.47:0/1878019089 learned_addr learned my addr 172.21.15.47:0/1878019089 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:17:16.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.436+0000 7fb9337fe640 10 client.?.objecter ms_handle_connect 0x7fb94c0ac5a0 2024-03-29T11:17:16.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.436+0000 7fb9337fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:16.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.436+0000 7fb9337fe640 10 client.?.objecter ms_handle_connect 0x7fb94c0b0770 2024-03-29T11:17:16.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.436+0000 7fb9337fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:16.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.436+0000 7fb9337fe640 1 -- 172.21.15.47:0/1878019089 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1823767094 0 0) 0x7fb94c145bb0 con 0x7fb94c0ac5a0 2024-03-29T11:17:16.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.436+0000 7fb9337fe640 10 monclient(hunting): _negotiate 2024-03-29T11:17:16.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.436+0000 7fb9337fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:16.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.436+0000 7fb9337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:16.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.436+0000 7fb9337fe640 10 monclient(hunting): my global_id is 4387 2024-03-29T11:17:16.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.436+0000 7fb9337fe640 1 -- 172.21.15.47:0/1878019089 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb938004eb0 con 0x7fb94c0ac5a0 2024-03-29T11:17:16.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.436+0000 7fb9337fe640 10 client.?.objecter ms_handle_connect 0x7fb94c14a700 2024-03-29T11:17:16.421 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.436+0000 7fb9337fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:16.421 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.436+0000 7fb9337fe640 1 -- 172.21.15.47:0/1878019089 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2459169511 0 0) 0x7fb94c126570 con 0x7fb94c0b0770 2024-03-29T11:17:16.421 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.436+0000 7fb9337fe640 10 monclient(hunting): _negotiate 2024-03-29T11:17:16.421 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.436+0000 7fb9337fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:16.421 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.436+0000 7fb9337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:16.421 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.436+0000 7fb9337fe640 10 monclient(hunting): my global_id is 4374 2024-03-29T11:17:16.421 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.436+0000 7fb9337fe640 1 -- 172.21.15.47:0/1878019089 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb94c145bb0 con 0x7fb94c0b0770 2024-03-29T11:17:16.421 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.436+0000 7fb9337fe640 1 -- 172.21.15.47:0/1878019089 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1052169826 0 0) 0x7fb938004eb0 con 0x7fb94c0ac5a0 2024-03-29T11:17:16.421 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.436+0000 7fb9337fe640 1 -- 172.21.15.47:0/1878019089 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb94c126570 con 0x7fb94c0ac5a0 2024-03-29T11:17:16.421 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.436+0000 7fb9337fe640 1 -- 172.21.15.47:0/1878019089 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fb93c0013b0 con 0x7fb94c0ac5a0 2024-03-29T11:17:16.421 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.436+0000 7fb9337fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:16.421 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.436+0000 7fb9337fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:17:16.421 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.436+0000 7fb9337fe640 10 monclient(hunting): dump: 2024-03-29T11:17:16.421 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:17:16.421 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:17:16.421 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:16.422 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:16.422 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:16.422 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:17:16.422 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:17:16.422 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:17:16.422 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:17:16.422 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:17:16.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.436+0000 7fb9337fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:17:16.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.436+0000 7fb9337fe640 1 -- 172.21.15.47:0/1878019089 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1561829697 0 0) 0x7fb94c1286b0 con 0x7fb94c14a700 2024-03-29T11:17:16.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.437+0000 7fb9337fe640 10 monclient(hunting): _negotiate 2024-03-29T11:17:16.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.437+0000 7fb9337fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:16.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.437+0000 7fb9337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:16.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.437+0000 7fb9337fe640 10 monclient(hunting): my global_id is 4412 2024-03-29T11:17:16.423 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.437+0000 7fb9337fe640 1 -- 172.21.15.47:0/1878019089 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb938004eb0 con 0x7fb94c14a700 2024-03-29T11:17:16.423 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.437+0000 7fb9337fe640 1 -- 172.21.15.47:0/1878019089 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 587857507 0 0) 0x7fb94c145bb0 con 0x7fb94c0b0770 2024-03-29T11:17:16.423 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.437+0000 7fb9337fe640 1 -- 172.21.15.47:0/1878019089 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb94c1286b0 con 0x7fb94c0b0770 2024-03-29T11:17:16.423 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.437+0000 7fb9337fe640 1 -- 172.21.15.47:0/1878019089 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fb9400018d0 con 0x7fb94c0b0770 2024-03-29T11:17:16.423 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.437+0000 7fb9337fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:16.423 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.437+0000 7fb9337fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:17:16.423 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.437+0000 7fb9337fe640 10 monclient(hunting): dump: 2024-03-29T11:17:16.423 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:17:16.423 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:17:16.423 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:16.423 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:16.423 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:16.423 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:17:16.423 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:17:16.423 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:17:16.424 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:17:16.424 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:17:16.424 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.437+0000 7fb9337fe640 1 -- 172.21.15.47:0/1878019089 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 239217533 0 0) 0x7fb94c126570 con 0x7fb94c0ac5a0 2024-03-29T11:17:16.424 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.437+0000 7fb9337fe640 1 -- 172.21.15.47:0/1878019089 >> v1:172.21.15.47:6790/0 conn(0x7fb94c14a700 legacy=0x7fb94c1454a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:16.424 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.437+0000 7fb9337fe640 1 -- 172.21.15.47:0/1878019089 >> v1:172.21.15.47:6789/0 conn(0x7fb94c0b0770 legacy=0x7fb94c144d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:16.424 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.437+0000 7fb9337fe640 10 monclient: _finish_hunting 0 2024-03-29T11:17:16.424 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.437+0000 7fb9337fe640 1 monclient: found mon.b 2024-03-29T11:17:16.424 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.437+0000 7fb9337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:16.424 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.437+0000 7fb9337fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:17:16.424 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.437+0000 7fb9337fe640 1 -- 172.21.15.47:0/1878019089 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb94c145e60 con 0x7fb94c0ac5a0 2024-03-29T11:17:16.424 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.437+0000 7fb9337fe640 10 monclient: _finish_auth 0 2024-03-29T11:17:16.424 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.437+0000 7fb9337fe640 10 monclient: _check_auth_tickets 2024-03-29T11:17:16.424 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.437+0000 7fb9337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:16.424 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.437+0000 7fb95142a640 5 monclient: authenticate success, global_id 4387 2024-03-29T11:17:16.424 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.437+0000 7fb95142a640 10 monclient: _renew_subs 2024-03-29T11:17:16.424 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.437+0000 7fb95142a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:17:16.425 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.437+0000 7fb95142a640 1 -- 172.21.15.47:0/1878019089 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb94c126050 con 0x7fb94c0ac5a0 2024-03-29T11:17:16.425 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.437+0000 7fb95142a640 10 client.4387.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:17:16.425 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.437+0000 7fb95142a640 10 monclient: _renew_subs 2024-03-29T11:17:16.425 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.437+0000 7fb95142a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:17:16.425 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.437+0000 7fb95142a640 1 -- 172.21.15.47:0/1878019089 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb94c1dc320 con 0x7fb94c0ac5a0 2024-03-29T11:17:16.425 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.437+0000 7fb95142a640 1 librados: init done 2024-03-29T11:17:16.425 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.438+0000 7fb9337fe640 1 -- 172.21.15.47:0/1878019089 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb93c0047e0 con 0x7fb94c0ac5a0 2024-03-29T11:17:16.425 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.438+0000 7fb9337fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:16.425 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.438+0000 7fb9337fe640 1 -- 172.21.15.47:0/1878019089 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fb93c004af0 con 0x7fb94c0ac5a0 2024-03-29T11:17:16.425 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.438+0000 7fb9337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:16.425 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.438+0000 7fb9337fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:17:16.425 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.438+0000 7fb9337fe640 10 monclient: dump: 2024-03-29T11:17:16.425 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:17:16.425 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:17:16.425 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:16.425 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:16.426 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:16.426 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:17:16.426 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:17:16.426 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:17:16.426 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:17:16.426 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:17:16.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.438+0000 7fb9497fa640 4 set_mon_vals no callback set 2024-03-29T11:17:16.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.439+0000 7fb95142a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:16.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.439+0000 7fb95142a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:16.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.439+0000 7fb95142a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:17:16.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.439+0000 7fb95142a640 1 -- 172.21.15.47:0/1878019089 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb92c005080 con 0x7fb94c0ac5a0 2024-03-29T11:17:16.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.440+0000 7fb9337fe640 1 -- 172.21.15.47:0/1878019089 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7fb93c01aac0 con 0x7fb94c0ac5a0 2024-03-29T11:17:16.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.440+0000 7fb9337fe640 1 -- 172.21.15.47:0/1878019089 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7fb93c084ac0 con 0x7fb94c0ac5a0 2024-03-29T11:17:16.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.440+0000 7fb9337fe640 10 client.4387.objecter ms_dispatch 0x7fb94c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:17:16.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.440+0000 7fb9337fe640 3 client.4387.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:17:16.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.440+0000 7fb9337fe640 3 client.4387.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:17:16.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.440+0000 7fb9337fe640 20 client.4387.objecter dump_active .. 0 homeless 2024-03-29T11:17:16.428 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.442+0000 7fb94a7fc640 10 monclient: get_auth_request con 0x7fb93806e140 auth_method 0 2024-03-29T11:17:16.428 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.442+0000 7fb9337fe640 10 client.4387.objecter ms_handle_connect 0x7fb93806e140 2024-03-29T11:17:16.428 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.449+0000 7fb9337fe640 1 -- 172.21.15.47:0/1878019089 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb93c0501d0 con 0x7fb94c0ac5a0 2024-03-29T11:17:16.428 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.449+0000 7fb9337fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:16.428 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.449+0000 7fb9337fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:17:16.817 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.840+0000 7fb95142a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:16.817 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.840+0000 7fb95142a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:16.817 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.840+0000 7fb95142a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:17:16.817 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.840+0000 7fb95142a640 1 -- 172.21.15.47:0/1878019089 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb92c005330 con 0x7fb94c0ac5a0 2024-03-29T11:17:16.820 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.843+0000 7fb9337fe640 1 -- 172.21.15.47:0/1878019089 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7fb93c0522d0 con 0x7fb94c0ac5a0 2024-03-29T11:17:16.820 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.843+0000 7fb9337fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:16.820 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.843+0000 7fb9337fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:17:16.820 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:17:16.820 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:17:16.823 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.846+0000 7fb95142a640 10 librados: watch_flush enter 2024-03-29T11:17:16.823 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.846+0000 7fb95142a640 10 librados: watch_flush exit 2024-03-29T11:17:16.823 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.846+0000 7fb95142a640 20 client.4387.objecter shutdown clearing up homeless session... 2024-03-29T11:17:16.823 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.846+0000 7fb95142a640 10 client.4387.objecter successfully canceled tick 2024-03-29T11:17:16.823 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.846+0000 7fb95142a640 1 -- 172.21.15.47:0/1878019089 >> v1:172.21.15.121:6816/1964117370 conn(0x7fb93806e140 legacy=0x7fb938070600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:16.824 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.846+0000 7fb95142a640 10 monclient: shutdown 2024-03-29T11:17:16.824 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.846+0000 7fb95142a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:16.824 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.846+0000 7fb95142a640 1 -- 172.21.15.47:0/1878019089 >> v1:172.21.15.121:6789/0 conn(0x7fb94c0ac5a0 legacy=0x7fb94c126a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:16.824 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.846+0000 7fb95142a640 1 -- 172.21.15.47:0/1878019089 shutdown_connections 2024-03-29T11:17:16.825 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.846+0000 7fb95142a640 1 -- 172.21.15.47:0/1878019089 >> 172.21.15.47:0/1878019089 conn(0x7fb94c06b660 msgr2=0x7fb94c0abe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:16.825 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.847+0000 7fb95142a640 1 -- 172.21.15.47:0/1878019089 shutdown_connections 2024-03-29T11:17:16.825 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.847+0000 7fb95142a640 1 -- 172.21.15.47:0/1878019089 wait complete. 2024-03-29T11:17:16.825 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:16.847+0000 7fb95142a640 1 librados: shutdown 2024-03-29T11:17:16.840 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:17:23.647 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:17:23.800 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.823+0000 7f0cf7c9a640 10 monclient: get_monmap_and_config 2024-03-29T11:17:23.800 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.823+0000 7f0cf7c9a640 10 monclient: build_initial_monmap 2024-03-29T11:17:23.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.823+0000 7f0cf7c9a640 10 monclient: monmap: 2024-03-29T11:17:23.801 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:17:23.801 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:17:23.801 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:17:23.824058+0000 2024-03-29T11:17:23.801 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:17:23.824058+0000 2024-03-29T11:17:23.801 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:23.801 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:17:23.801 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:17:23.801 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:17:23.801 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:17:23.801 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:17:23.804 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.823+0000 7f0cf7c9a640 1 Processor -- start 2024-03-29T11:17:23.804 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.824+0000 7f0cf7c9a640 1 -- start start 2024-03-29T11:17:23.804 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.824+0000 7f0cf7c9a640 10 monclient: init 2024-03-29T11:17:23.804 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.825+0000 7f0cf7c9a640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:23.804 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.825+0000 7f0cf7c9a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:17:23.804 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.827+0000 7f0cf5a36640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f0cf00b4330 0x7f0cf00b4710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:38063/0 (socket says 172.21.15.47:38063) 2024-03-29T11:17:23.804 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.827+0000 7f0cf5a36640 1 -- 172.21.15.47:0/3297814205 learned_addr learned my addr 172.21.15.47:0/3297814205 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:17:23.804 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.827+0000 7f0cf7c9a640 10 monclient(hunting): picked mon.noname-a con 0x7f0cf00b4330 addr v1:172.21.15.47:6789/0 2024-03-29T11:17:23.804 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.827+0000 7f0cf7c9a640 10 monclient(hunting): picked mon.noname-b con 0x7f0cf01470b0 addr v1:172.21.15.47:6790/0 2024-03-29T11:17:23.805 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.827+0000 7f0cf7c9a640 10 monclient(hunting): picked mon.noname-c con 0x7f0cf01439b0 addr v1:172.21.15.121:6789/0 2024-03-29T11:17:23.805 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.827+0000 7f0cf7c9a640 1 -- 172.21.15.47:0/3297814205 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cf006f2a0 con 0x7f0cf00b4330 2024-03-29T11:17:23.805 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.827+0000 7f0cf7c9a640 1 -- 172.21.15.47:0/3297814205 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cf0146b90 con 0x7f0cf01439b0 2024-03-29T11:17:23.805 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.827+0000 7f0cf7c9a640 1 -- 172.21.15.47:0/3297814205 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cf00b2ca0 con 0x7f0cf01470b0 2024-03-29T11:17:23.805 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.827+0000 7f0cf7c9a640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:23.805 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.827+0000 7f0cf7c9a640 10 monclient(hunting): authenticate will time out at 1087.015619s 2024-03-29T11:17:23.805 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.827+0000 7f0cf4a34640 1 -- 172.21.15.47:0/3297814205 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1186561893 0 0) 0x7f0cf006f2a0 con 0x7f0cf00b4330 2024-03-29T11:17:23.806 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.827+0000 7f0cf4a34640 10 monclient(hunting): _negotiate 2024-03-29T11:17:23.806 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.827+0000 7f0cf4a34640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:23.806 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.827+0000 7f0cf4a34640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:23.806 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.827+0000 7f0cf4a34640 10 monclient(hunting): my global_id is 4377 2024-03-29T11:17:23.806 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.827+0000 7f0cf4a34640 1 -- 172.21.15.47:0/3297814205 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cdc003520 con 0x7f0cf00b4330 2024-03-29T11:17:23.806 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.828+0000 7f0cf4a34640 1 -- 172.21.15.47:0/3297814205 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3625090877 0 0) 0x7f0cdc003520 con 0x7f0cf00b4330 2024-03-29T11:17:23.806 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.828+0000 7f0cf4a34640 1 -- 172.21.15.47:0/3297814205 >> v1:172.21.15.47:6790/0 conn(0x7f0cf01470b0 legacy=0x7f0cf00b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:17:23.806 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.828+0000 7f0cf4a34640 1 -- 172.21.15.47:0/3297814205 >> v1:172.21.15.121:6789/0 conn(0x7f0cf01439b0 legacy=0x7f0cf0151d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:17:23.807 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.830+0000 7f0cf4a34640 10 monclient: _finish_hunting 0 2024-03-29T11:17:23.807 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.830+0000 7f0cf4a34640 1 monclient: found mon.noname-a 2024-03-29T11:17:23.807 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.830+0000 7f0cf4a34640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:23.807 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.830+0000 7f0cf4a34640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-29T11:17:23.808 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.830+0000 7f0cf4a34640 1 -- 172.21.15.47:0/3297814205 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cf006e0c0 con 0x7f0cf00b4330 2024-03-29T11:17:23.808 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.830+0000 7f0cf4a34640 10 monclient: _finish_auth 0 2024-03-29T11:17:23.808 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.830+0000 7f0cf4a34640 10 monclient: _check_auth_tickets 2024-03-29T11:17:23.808 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.830+0000 7f0cf4a34640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:23.808 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.830+0000 7f0cf4a34640 1 -- 172.21.15.47:0/3297814205 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f0cec001690 con 0x7f0cf00b4330 2024-03-29T11:17:23.808 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.830+0000 7f0cf4a34640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:23.808 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.830+0000 7f0cf4a34640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:17:23.808 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.830+0000 7f0cf4a34640 10 monclient: dump: 2024-03-29T11:17:23.808 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:17:23.808 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:17:23.808 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:23.808 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:23.808 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:23.808 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:17:23.808 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:17:23.809 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:17:23.809 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:17:23.809 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:17:23.809 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.830+0000 7f0cf7c9a640 5 monclient: authenticate success, global_id 4377 2024-03-29T11:17:23.809 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.830+0000 7f0cf7c9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:17:23.810 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.831+0000 7f0cf4a34640 1 -- 172.21.15.47:0/3297814205 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0cec001860 con 0x7f0cf00b4330 2024-03-29T11:17:23.811 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.831+0000 7f0cf4a34640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:23.811 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.831+0000 7f0cf4a34640 1 -- 172.21.15.47:0/3297814205 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f0cec001cb0 con 0x7f0cf00b4330 2024-03-29T11:17:23.811 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.831+0000 7f0cf4a34640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:23.811 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.831+0000 7f0cf4a34640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:17:23.811 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.831+0000 7f0cf4a34640 10 monclient: dump: 2024-03-29T11:17:23.811 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:17:23.811 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:17:23.811 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:23.811 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:23.811 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:23.812 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:17:23.812 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:17:23.812 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:17:23.812 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:17:23.812 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:17:23.812 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.831+0000 7f0cf7c9a640 10 monclient: get_monmap_and_config success 2024-03-29T11:17:23.812 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.831+0000 7f0cf7c9a640 4 set_mon_vals no callback set 2024-03-29T11:17:23.812 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.831+0000 7f0cf7c9a640 10 monclient: shutdown 2024-03-29T11:17:23.812 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.831+0000 7f0cf7c9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:23.812 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.831+0000 7f0cf7c9a640 1 -- 172.21.15.47:0/3297814205 >> v1:172.21.15.47:6789/0 conn(0x7f0cf00b4330 legacy=0x7f0cf00b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:23.812 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.832+0000 7f0cf7c9a640 1 -- 172.21.15.47:0/3297814205 shutdown_connections 2024-03-29T11:17:23.812 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.832+0000 7f0cf7c9a640 1 -- 172.21.15.47:0/3297814205 >> 172.21.15.47:0/3297814205 conn(0x7f0cf0066a00 msgr2=0x7f0cf0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:23.812 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.832+0000 7f0cf7c9a640 1 -- 172.21.15.47:0/3297814205 shutdown_connections 2024-03-29T11:17:23.814 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.832+0000 7f0cf7c9a640 1 -- 172.21.15.47:0/3297814205 wait complete. 2024-03-29T11:17:23.814 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.833+0000 7f0cf7c9a640 10 monclient: build_initial_monmap 2024-03-29T11:17:23.814 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.833+0000 7f0cf7c9a640 10 monclient: monmap: 2024-03-29T11:17:23.814 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:17:23.814 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:17:23.814 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:17:23.814 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:17:23.814 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:23.814 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:17:23.814 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:17:23.814 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:17:23.815 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:17:23.815 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:17:23.815 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.833+0000 7f0cf7c9a640 1 librados: starting msgr at 2024-03-29T11:17:23.815 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.833+0000 7f0cf7c9a640 1 librados: starting objecter 2024-03-29T11:17:23.815 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.833+0000 7f0cf7c9a640 1 Processor -- start 2024-03-29T11:17:23.815 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.834+0000 7f0cf7c9a640 1 -- start start 2024-03-29T11:17:23.815 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.834+0000 7f0cf7c9a640 1 librados: setting wanted keys 2024-03-29T11:17:23.815 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.834+0000 7f0cf7c9a640 1 librados: calling monclient init 2024-03-29T11:17:23.815 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.834+0000 7f0cf7c9a640 10 monclient: init 2024-03-29T11:17:23.815 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.834+0000 7f0cf7c9a640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:23.815 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.834+0000 7f0cf7c9a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:17:23.815 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.835+0000 7f0cf7c9a640 10 monclient(hunting): picked mon.noname-b con 0x7f0cf00b4330 addr v1:172.21.15.121:6789/0 2024-03-29T11:17:23.815 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.835+0000 7f0cf7c9a640 10 monclient(hunting): picked mon.noname-c con 0x7f0cf01439b0 addr v1:172.21.15.47:6790/0 2024-03-29T11:17:23.815 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.835+0000 7f0cf7c9a640 10 monclient(hunting): picked mon.noname-a con 0x7f0cf01470b0 addr v1:172.21.15.47:6789/0 2024-03-29T11:17:23.816 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.835+0000 7f0cf7c9a640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cf0120970 con 0x7f0cf01470b0 2024-03-29T11:17:23.816 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.835+0000 7f0cf7c9a640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cf01416c0 con 0x7f0cf00b4330 2024-03-29T11:17:23.816 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.835+0000 7f0cf7c9a640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cf0146440 con 0x7f0cf01439b0 2024-03-29T11:17:23.816 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.835+0000 7f0cf7c9a640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:23.816 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.835+0000 7f0cf7c9a640 10 monclient(hunting): authenticate will time out at 1087.023631s 2024-03-29T11:17:23.816 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.836+0000 7f0cf5235640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f0cf01439b0 0x7f0cf01404a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:50166/0 (socket says 172.21.15.47:50166) 2024-03-29T11:17:23.816 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.836+0000 7f0cf5235640 1 -- 172.21.15.47:0/3905622503 learned_addr learned my addr 172.21.15.47:0/3905622503 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:17:23.816 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.836+0000 7f0ce67fc640 10 client.?.objecter ms_handle_connect 0x7f0cf01439b0 2024-03-29T11:17:23.816 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.836+0000 7f0ce67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:23.816 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.836+0000 7f0ce67fc640 1 -- 172.21.15.47:0/3905622503 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1487954738 0 0) 0x7f0cf0146440 con 0x7f0cf01439b0 2024-03-29T11:17:23.816 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.837+0000 7f0ce67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:17:23.816 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.837+0000 7f0ce67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:23.816 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.837+0000 7f0ce67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:23.816 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.837+0000 7f0ce67fc640 10 monclient(hunting): my global_id is 4415 2024-03-29T11:17:23.816 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.837+0000 7f0ce67fc640 1 -- 172.21.15.47:0/3905622503 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cd4004eb0 con 0x7f0cf01439b0 2024-03-29T11:17:23.819 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.837+0000 7f0ce67fc640 1 -- 172.21.15.47:0/3905622503 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2430378180 0 0) 0x7f0cd4004eb0 con 0x7f0cf01439b0 2024-03-29T11:17:23.819 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.837+0000 7f0ce67fc640 1 -- 172.21.15.47:0/3905622503 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0cf0146440 con 0x7f0cf01439b0 2024-03-29T11:17:23.819 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.837+0000 7f0ce67fc640 1 -- 172.21.15.47:0/3905622503 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f0ce0001960 con 0x7f0cf01439b0 2024-03-29T11:17:23.819 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.837+0000 7f0ce67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:23.819 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.837+0000 7f0ce67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:17:23.819 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.837+0000 7f0ce67fc640 10 monclient(hunting): dump: 2024-03-29T11:17:23.819 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:17:23.819 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:17:23.820 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:23.820 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:23.820 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:23.820 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:17:23.820 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:17:23.820 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:17:23.820 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:17:23.820 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:17:23.820 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.837+0000 7f0ce67fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:17:23.820 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.838+0000 7f0ce67fc640 1 -- 172.21.15.47:0/3905622503 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1287444753 0 0) 0x7f0cf0146440 con 0x7f0cf01439b0 2024-03-29T11:17:23.820 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.838+0000 7f0ce67fc640 1 -- 172.21.15.47:0/3905622503 >> v1:172.21.15.121:6789/0 conn(0x7f0cf00b4330 legacy=0x7f0cf0120260 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:17:23.820 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.838+0000 7f0ce67fc640 1 -- 172.21.15.47:0/3905622503 >> v1:172.21.15.47:6789/0 conn(0x7f0cf01470b0 legacy=0x7f0cf01e6d60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:17:23.820 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.839+0000 7f0ce67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:17:23.820 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.839+0000 7f0ce67fc640 1 monclient: found mon.c 2024-03-29T11:17:23.820 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.839+0000 7f0ce67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:23.821 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.839+0000 7f0ce67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:17:23.821 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.839+0000 7f0ce67fc640 1 -- 172.21.15.47:0/3905622503 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cf01465d0 con 0x7f0cf01439b0 2024-03-29T11:17:23.821 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.839+0000 7f0ce67fc640 10 monclient: _finish_auth 0 2024-03-29T11:17:23.821 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.839+0000 7f0ce67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:17:23.821 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.839+0000 7f0ce67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:23.821 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.840+0000 7f0cf7c9a640 5 monclient: authenticate success, global_id 4415 2024-03-29T11:17:23.821 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.840+0000 7f0cf7c9a640 10 monclient: _renew_subs 2024-03-29T11:17:23.821 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.840+0000 7f0cf7c9a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:17:23.821 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.840+0000 7f0cf7c9a640 1 -- 172.21.15.47:0/3905622503 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cf011f770 con 0x7f0cf01439b0 2024-03-29T11:17:23.821 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.840+0000 7f0ce67fc640 1 -- 172.21.15.47:0/3905622503 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0ce0001ac0 con 0x7f0cf01439b0 2024-03-29T11:17:23.821 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.840+0000 7f0ce67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:23.821 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.840+0000 7f0cf7c9a640 10 client.4415.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:17:23.821 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.840+0000 7f0cf7c9a640 10 monclient: _renew_subs 2024-03-29T11:17:23.821 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.840+0000 7f0ce67fc640 1 -- 172.21.15.47:0/3905622503 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f0ce0003a40 con 0x7f0cf01439b0 2024-03-29T11:17:23.821 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.840+0000 7f0cf7c9a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:17:23.822 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.840+0000 7f0cf7c9a640 1 -- 172.21.15.47:0/3905622503 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cf00b7570 con 0x7f0cf01439b0 2024-03-29T11:17:23.822 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.840+0000 7f0ce7fff640 4 set_mon_vals no callback set 2024-03-29T11:17:23.822 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.840+0000 7f0cf7c9a640 1 librados: init done 2024-03-29T11:17:23.822 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.840+0000 7f0ce67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:23.822 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.840+0000 7f0ce67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:17:23.822 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.840+0000 7f0ce67fc640 10 monclient: dump: 2024-03-29T11:17:23.822 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:17:23.822 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:17:23.822 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:23.822 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:23.822 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:23.822 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:17:23.822 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:17:23.822 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:17:23.823 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:17:23.823 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:17:23.823 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.841+0000 7f0cd3fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:23.823 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.842+0000 7f0cd3fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:23.823 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.842+0000 7f0cd3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:17:23.823 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.842+0000 7f0cd3fff640 1 -- 172.21.15.47:0/3905622503 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cf01217e0 con 0x7f0cf01439b0 2024-03-29T11:17:23.823 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.842+0000 7f0ce67fc640 1 -- 172.21.15.47:0/3905622503 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7f0ce0003c20 con 0x7f0cf01439b0 2024-03-29T11:17:23.824 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.843+0000 7f0cf5a36640 10 monclient: get_auth_request con 0x7f0cd406db30 auth_method 0 2024-03-29T11:17:23.824 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.843+0000 7f0ce67fc640 1 -- 172.21.15.47:0/3905622503 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7f0ce0084b90 con 0x7f0cf01439b0 2024-03-29T11:17:23.824 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.843+0000 7f0ce67fc640 10 client.4415.objecter ms_dispatch 0x7f0cf0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:17:23.824 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.843+0000 7f0ce67fc640 3 client.4415.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:17:23.824 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.843+0000 7f0ce67fc640 3 client.4415.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:17:23.824 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.843+0000 7f0ce67fc640 20 client.4415.objecter dump_active .. 0 homeless 2024-03-29T11:17:23.824 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.843+0000 7f0ce67fc640 10 client.4415.objecter ms_handle_connect 0x7f0cd406db30 2024-03-29T11:17:23.827 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.850+0000 7f0ce67fc640 1 -- 172.21.15.47:0/3905622503 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0ce0050470 con 0x7f0cf01439b0 2024-03-29T11:17:23.827 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.850+0000 7f0ce67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:23.827 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:23.850+0000 7f0ce67fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:17:24.162 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:24.185+0000 7f0cd3fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:24.163 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:24.185+0000 7f0cd3fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:24.163 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:24.185+0000 7f0cd3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:17:24.163 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:24.185+0000 7f0cd3fff640 1 -- 172.21.15.47:0/3905622503 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0cf011ff50 con 0x7f0cf01439b0 2024-03-29T11:17:24.165 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:24.186+0000 7f0ce67fc640 1 -- 172.21.15.47:0/3905622503 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7f0ce00523a0 con 0x7f0cf01439b0 2024-03-29T11:17:24.165 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:24.186+0000 7f0ce67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:24.165 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:24.186+0000 7f0ce67fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:17:24.165 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:17:24.165 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:17:24.168 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:24.191+0000 7f0cd3fff640 10 librados: watch_flush enter 2024-03-29T11:17:24.169 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:24.191+0000 7f0cd3fff640 10 librados: watch_flush exit 2024-03-29T11:17:24.169 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:24.191+0000 7f0cd3fff640 20 client.4415.objecter shutdown clearing up homeless session... 2024-03-29T11:17:24.169 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:24.191+0000 7f0cd3fff640 10 client.4415.objecter successfully canceled tick 2024-03-29T11:17:24.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:24.193+0000 7f0cd3fff640 1 -- 172.21.15.47:0/3905622503 >> v1:172.21.15.121:6816/1964117370 conn(0x7f0cd406db30 legacy=0x7f0cd406fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:24.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:24.193+0000 7f0cd3fff640 10 monclient: shutdown 2024-03-29T11:17:24.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:24.193+0000 7f0cd3fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:24.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:24.193+0000 7f0cd3fff640 1 -- 172.21.15.47:0/3905622503 >> v1:172.21.15.47:6790/0 conn(0x7f0cf01439b0 legacy=0x7f0cf01404a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:24.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:24.196+0000 7f0cd3fff640 1 -- 172.21.15.47:0/3905622503 shutdown_connections 2024-03-29T11:17:24.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:24.196+0000 7f0cd3fff640 1 -- 172.21.15.47:0/3905622503 >> 172.21.15.47:0/3905622503 conn(0x7f0cf0066a00 msgr2=0x7f0cf01d1320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:24.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:24.197+0000 7f0cd3fff640 1 -- 172.21.15.47:0/3905622503 shutdown_connections 2024-03-29T11:17:24.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:24.197+0000 7f0cd3fff640 1 -- 172.21.15.47:0/3905622503 wait complete. 2024-03-29T11:17:24.175 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:24.198+0000 7f0cd3fff640 1 librados: shutdown 2024-03-29T11:17:24.191 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:17:30.999 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:17:31.150 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.173+0000 7fcc13fd6640 10 monclient: get_monmap_and_config 2024-03-29T11:17:31.151 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.173+0000 7fcc13fd6640 10 monclient: build_initial_monmap 2024-03-29T11:17:31.151 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.173+0000 7fcc13fd6640 10 monclient: monmap: 2024-03-29T11:17:31.151 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:17:31.151 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:17:31.151 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:17:31.174686+0000 2024-03-29T11:17:31.151 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:17:31.174686+0000 2024-03-29T11:17:31.151 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:31.151 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:17:31.151 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:17:31.151 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:17:31.151 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:17:31.151 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:17:31.153 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.175+0000 7fcc13fd6640 1 Processor -- start 2024-03-29T11:17:31.154 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.175+0000 7fcc13fd6640 1 -- start start 2024-03-29T11:17:31.154 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.175+0000 7fcc13fd6640 10 monclient: init 2024-03-29T11:17:31.154 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.176+0000 7fcc13fd6640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:31.154 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.176+0000 7fcc13fd6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:17:31.154 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.176+0000 7fcc13fd6640 10 monclient(hunting): picked mon.noname-b con 0x7fcc0c1457f0 addr v1:172.21.15.47:6790/0 2024-03-29T11:17:31.154 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.176+0000 7fcc13fd6640 10 monclient(hunting): picked mon.noname-a con 0x7fcc0c13f650 addr v1:172.21.15.47:6789/0 2024-03-29T11:17:31.154 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.176+0000 7fcc13fd6640 10 monclient(hunting): picked mon.noname-c con 0x7fcc0c0a9d70 addr v1:172.21.15.121:6789/0 2024-03-29T11:17:31.154 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.176+0000 7fcc13fd6640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc0c0aaef0 con 0x7fcc0c13f650 2024-03-29T11:17:31.155 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.176+0000 7fcc13fd6640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc0c145bd0 con 0x7fcc0c0a9d70 2024-03-29T11:17:31.155 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.176+0000 7fcc13fd6640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc0c145eb0 con 0x7fcc0c1457f0 2024-03-29T11:17:31.155 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.176+0000 7fcc13fd6640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:31.155 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.176+0000 7fcc13fd6640 10 monclient(hunting): authenticate will time out at 1094.364666s 2024-03-29T11:17:31.155 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.177+0000 7fcc11d72640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fcc0c1457f0 0x7fcc0c13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:42258/0 (socket says 172.21.15.47:42258) 2024-03-29T11:17:31.155 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.177+0000 7fcc11d72640 1 -- 172.21.15.47:0/2527402381 learned_addr learned my addr 172.21.15.47:0/2527402381 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:17:31.156 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.177+0000 7fcc10d70640 1 -- 172.21.15.47:0/2527402381 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3667917490 0 0) 0x7fcc0c145eb0 con 0x7fcc0c1457f0 2024-03-29T11:17:31.156 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.177+0000 7fcc10d70640 10 monclient(hunting): _negotiate 2024-03-29T11:17:31.156 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.177+0000 7fcc10d70640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:31.156 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.177+0000 7fcc10d70640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:31.156 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.177+0000 7fcc10d70640 10 monclient(hunting): my global_id is 4418 2024-03-29T11:17:31.156 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.177+0000 7fcc10d70640 1 -- 172.21.15.47:0/2527402381 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbf8003520 con 0x7fcc0c1457f0 2024-03-29T11:17:31.156 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.178+0000 7fcc10d70640 1 -- 172.21.15.47:0/2527402381 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4158063077 0 0) 0x7fcbf8003520 con 0x7fcc0c1457f0 2024-03-29T11:17:31.156 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.178+0000 7fcc10d70640 1 -- 172.21.15.47:0/2527402381 >> v1:172.21.15.121:6789/0 conn(0x7fcc0c0a9d70 legacy=0x7fcc0c14d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:17:31.160 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.180+0000 7fcc10d70640 1 -- 172.21.15.47:0/2527402381 >> v1:172.21.15.47:6789/0 conn(0x7fcc0c13f650 legacy=0x7fcc0c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:31.160 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.180+0000 7fcc10d70640 10 monclient: _finish_hunting 0 2024-03-29T11:17:31.160 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.180+0000 7fcc10d70640 1 monclient: found mon.noname-b 2024-03-29T11:17:31.160 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.180+0000 7fcc10d70640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:31.160 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.180+0000 7fcc10d70640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-29T11:17:31.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.180+0000 7fcc10d70640 1 -- 172.21.15.47:0/2527402381 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc0c146340 con 0x7fcc0c1457f0 2024-03-29T11:17:31.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.180+0000 7fcc10d70640 10 monclient: _finish_auth 0 2024-03-29T11:17:31.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.180+0000 7fcc10d70640 10 monclient: _check_auth_tickets 2024-03-29T11:17:31.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.180+0000 7fcc10d70640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:31.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.180+0000 7fcc10d70640 1 -- 172.21.15.47:0/2527402381 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fcc04001a10 con 0x7fcc0c1457f0 2024-03-29T11:17:31.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.180+0000 7fcc10d70640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:31.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.180+0000 7fcc10d70640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:17:31.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.180+0000 7fcc10d70640 10 monclient: dump: 2024-03-29T11:17:31.161 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:17:31.161 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:17:31.161 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:31.161 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:31.161 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:31.161 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:17:31.161 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:17:31.162 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:17:31.162 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:17:31.162 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:17:31.162 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.181+0000 7fcc13fd6640 5 monclient: authenticate success, global_id 4418 2024-03-29T11:17:31.162 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.181+0000 7fcc13fd6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:17:31.162 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.181+0000 7fcc10d70640 1 -- 172.21.15.47:0/2527402381 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcc04001be0 con 0x7fcc0c1457f0 2024-03-29T11:17:31.162 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.181+0000 7fcc10d70640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:31.162 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.181+0000 7fcc10d70640 1 -- 172.21.15.47:0/2527402381 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fcc04002030 con 0x7fcc0c1457f0 2024-03-29T11:17:31.162 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.181+0000 7fcc10d70640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:31.162 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.181+0000 7fcc10d70640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:17:31.162 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.181+0000 7fcc10d70640 10 monclient: dump: 2024-03-29T11:17:31.162 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:17:31.162 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:17:31.163 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:31.163 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:31.163 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:31.163 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:17:31.163 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:17:31.163 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:17:31.163 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:17:31.163 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:17:31.163 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.181+0000 7fcc13fd6640 10 monclient: get_monmap_and_config success 2024-03-29T11:17:31.163 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.181+0000 7fcc13fd6640 4 set_mon_vals no callback set 2024-03-29T11:17:31.163 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.182+0000 7fcc13fd6640 10 monclient: shutdown 2024-03-29T11:17:31.163 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.182+0000 7fcc13fd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:31.163 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.182+0000 7fcc13fd6640 1 -- 172.21.15.47:0/2527402381 >> v1:172.21.15.47:6790/0 conn(0x7fcc0c1457f0 legacy=0x7fcc0c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:31.163 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.182+0000 7fcc13fd6640 1 -- 172.21.15.47:0/2527402381 shutdown_connections 2024-03-29T11:17:31.163 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.182+0000 7fcc13fd6640 1 -- 172.21.15.47:0/2527402381 >> 172.21.15.47:0/2527402381 conn(0x7fcc0c066a00 msgr2=0x7fcc0c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:31.163 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.183+0000 7fcc13fd6640 1 -- 172.21.15.47:0/2527402381 shutdown_connections 2024-03-29T11:17:31.165 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.183+0000 7fcc13fd6640 1 -- 172.21.15.47:0/2527402381 wait complete. 2024-03-29T11:17:31.165 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.184+0000 7fcc13fd6640 10 monclient: build_initial_monmap 2024-03-29T11:17:31.165 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.184+0000 7fcc13fd6640 10 monclient: monmap: 2024-03-29T11:17:31.165 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:17:31.165 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:17:31.165 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:17:31.166 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:17:31.166 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:31.166 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:17:31.166 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:17:31.166 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:17:31.166 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:17:31.166 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:17:31.166 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.186+0000 7fcc13fd6640 1 librados: starting msgr at 2024-03-29T11:17:31.166 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.186+0000 7fcc13fd6640 1 librados: starting objecter 2024-03-29T11:17:31.166 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.186+0000 7fcc13fd6640 1 Processor -- start 2024-03-29T11:17:31.166 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.186+0000 7fcc13fd6640 1 -- start start 2024-03-29T11:17:31.166 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.186+0000 7fcc13fd6640 1 librados: setting wanted keys 2024-03-29T11:17:31.166 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.186+0000 7fcc13fd6640 1 librados: calling monclient init 2024-03-29T11:17:31.167 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.186+0000 7fcc13fd6640 10 monclient: init 2024-03-29T11:17:31.167 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.187+0000 7fcc13fd6640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:31.167 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.187+0000 7fcc13fd6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:17:31.167 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.187+0000 7fcc13fd6640 10 monclient(hunting): picked mon.noname-a con 0x7fcc0c0a9d70 addr v1:172.21.15.47:6789/0 2024-03-29T11:17:31.167 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.187+0000 7fcc13fd6640 10 monclient(hunting): picked mon.noname-b con 0x7fcc0c13f650 addr v1:172.21.15.121:6789/0 2024-03-29T11:17:31.167 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.187+0000 7fcc13fd6640 10 monclient(hunting): picked mon.noname-c con 0x7fcc0c1457f0 addr v1:172.21.15.47:6790/0 2024-03-29T11:17:31.167 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.187+0000 7fcc13fd6640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc0c13a990 con 0x7fcc0c0a9d70 2024-03-29T11:17:31.167 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.187+0000 7fcc13fd6640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc0c13ac70 con 0x7fcc0c13f650 2024-03-29T11:17:31.167 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.187+0000 7fcc13fd6640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc0c13af50 con 0x7fcc0c1457f0 2024-03-29T11:17:31.167 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.187+0000 7fcc13fd6640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:31.167 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.187+0000 7fcc13fd6640 10 monclient(hunting): authenticate will time out at 1094.375880s 2024-03-29T11:17:31.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.190+0000 7fcc12573640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fcc0c1457f0 0x7fcc0c1e64c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:42898/0 (socket says 172.21.15.47:42898) 2024-03-29T11:17:31.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.190+0000 7fcc12573640 1 -- 172.21.15.47:0/3937464133 learned_addr learned my addr 172.21.15.47:0/3937464133 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:17:31.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.190+0000 7fcc027fc640 10 client.?.objecter ms_handle_connect 0x7fcc0c1457f0 2024-03-29T11:17:31.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.190+0000 7fcc027fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:31.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.191+0000 7fcc027fc640 10 client.?.objecter ms_handle_connect 0x7fcc0c13f650 2024-03-29T11:17:31.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.191+0000 7fcc027fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:31.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.191+0000 7fcc027fc640 1 -- 172.21.15.47:0/3937464133 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3531951841 0 0) 0x7fcc0c13ac70 con 0x7fcc0c13f650 2024-03-29T11:17:31.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.191+0000 7fcc027fc640 10 monclient(hunting): _negotiate 2024-03-29T11:17:31.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.191+0000 7fcc027fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:31.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.191+0000 7fcc027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:31.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.191+0000 7fcc027fc640 10 monclient(hunting): my global_id is 4390 2024-03-29T11:17:31.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.191+0000 7fcc027fc640 1 -- 172.21.15.47:0/3937464133 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbf0004eb0 con 0x7fcc0c13f650 2024-03-29T11:17:31.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.192+0000 7fcc027fc640 1 -- 172.21.15.47:0/3937464133 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3009542703 0 0) 0x7fcbf0004eb0 con 0x7fcc0c13f650 2024-03-29T11:17:31.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.192+0000 7fcc027fc640 1 -- 172.21.15.47:0/3937464133 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcc0c13ac70 con 0x7fcc0c13f650 2024-03-29T11:17:31.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.192+0000 7fcc027fc640 1 -- 172.21.15.47:0/3937464133 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fcc080018c0 con 0x7fcc0c13f650 2024-03-29T11:17:31.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.192+0000 7fcc027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:31.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.192+0000 7fcc027fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:17:31.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.192+0000 7fcc027fc640 10 monclient(hunting): dump: 2024-03-29T11:17:31.172 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:17:31.172 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:17:31.173 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:31.173 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:31.173 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:31.173 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:17:31.173 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:17:31.173 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:17:31.173 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:17:31.173 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:17:31.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.192+0000 7fcc027fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:17:31.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.192+0000 7fcc027fc640 1 -- 172.21.15.47:0/3937464133 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3146518710 0 0) 0x7fcc0c13ac70 con 0x7fcc0c13f650 2024-03-29T11:17:31.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.192+0000 7fcc027fc640 1 -- 172.21.15.47:0/3937464133 >> v1:172.21.15.47:6790/0 conn(0x7fcc0c1457f0 legacy=0x7fcc0c1e64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:31.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.192+0000 7fcc027fc640 1 -- 172.21.15.47:0/3937464133 >> v1:172.21.15.47:6789/0 conn(0x7fcc0c0a9d70 legacy=0x7fcc0c13b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:31.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.192+0000 7fcc027fc640 10 monclient: _finish_hunting 0 2024-03-29T11:17:31.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.192+0000 7fcc027fc640 1 monclient: found mon.b 2024-03-29T11:17:31.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.192+0000 7fcc027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:31.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.192+0000 7fcc027fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:17:31.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.192+0000 7fcc027fc640 1 -- 172.21.15.47:0/3937464133 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc0c1417f0 con 0x7fcc0c13f650 2024-03-29T11:17:31.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.193+0000 7fcc027fc640 10 monclient: _finish_auth 0 2024-03-29T11:17:31.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.193+0000 7fcc027fc640 10 monclient: _check_auth_tickets 2024-03-29T11:17:31.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.193+0000 7fcc027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:31.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.193+0000 7fcc13fd6640 5 monclient: authenticate success, global_id 4390 2024-03-29T11:17:31.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.193+0000 7fcc13fd6640 10 monclient: _renew_subs 2024-03-29T11:17:31.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.193+0000 7fcc13fd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:17:31.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.193+0000 7fcc13fd6640 1 -- 172.21.15.47:0/3937464133 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc0c071f00 con 0x7fcc0c13f650 2024-03-29T11:17:31.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.193+0000 7fcc13fd6640 10 client.4390.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:17:31.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.193+0000 7fcc13fd6640 10 monclient: _renew_subs 2024-03-29T11:17:31.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.193+0000 7fcc13fd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:17:31.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.193+0000 7fcc13fd6640 1 -- 172.21.15.47:0/3937464133 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc0c13ba80 con 0x7fcc0c13f650 2024-03-29T11:17:31.175 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.193+0000 7fcc13fd6640 1 librados: init done 2024-03-29T11:17:31.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.195+0000 7fcc027fc640 1 -- 172.21.15.47:0/3937464133 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcc080035f0 con 0x7fcc0c13f650 2024-03-29T11:17:31.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.196+0000 7fcc027fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:31.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.196+0000 7fcc13fd6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:31.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.196+0000 7fcc027fc640 1 -- 172.21.15.47:0/3937464133 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fcc080039c0 con 0x7fcc0c13f650 2024-03-29T11:17:31.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.196+0000 7fcc027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:31.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.196+0000 7fcc027fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:17:31.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.196+0000 7fcc027fc640 10 monclient: dump: 2024-03-29T11:17:31.177 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:17:31.177 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:17:31.177 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:31.177 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:31.177 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:31.177 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:17:31.177 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:17:31.177 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:17:31.177 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:17:31.177 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:17:31.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.196+0000 7fcc03fff640 4 set_mon_vals no callback set 2024-03-29T11:17:31.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.197+0000 7fcc027fc640 1 -- 172.21.15.47:0/3937464133 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7fcc08003be0 con 0x7fcc0c13f650 2024-03-29T11:17:31.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.198+0000 7fcc027fc640 1 -- 172.21.15.47:0/3937464133 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7fcc080838f0 con 0x7fcc0c13f650 2024-03-29T11:17:31.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.198+0000 7fcc027fc640 10 client.4390.objecter ms_dispatch 0x7fcc0c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:17:31.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.198+0000 7fcc027fc640 3 client.4390.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:17:31.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.198+0000 7fcc027fc640 3 client.4390.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:17:31.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.198+0000 7fcc027fc640 20 client.4390.objecter dump_active .. 0 homeless 2024-03-29T11:17:31.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.200+0000 7fcc13fd6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:31.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.200+0000 7fcc13fd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:17:31.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.200+0000 7fcc13fd6640 1 -- 172.21.15.47:0/3937464133 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc0c142e00 con 0x7fcc0c13f650 2024-03-29T11:17:31.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.200+0000 7fcc11d72640 10 monclient: get_auth_request con 0x7fcbf006dcb0 auth_method 0 2024-03-29T11:17:31.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.200+0000 7fcc027fc640 10 client.4390.objecter ms_handle_connect 0x7fcbf006dcb0 2024-03-29T11:17:31.186 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.209+0000 7fcc027fc640 1 -- 172.21.15.47:0/3937464133 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcc0804f290 con 0x7fcc0c13f650 2024-03-29T11:17:31.186 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.209+0000 7fcc027fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:31.186 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.209+0000 7fcc027fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:17:31.512 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.534+0000 7fcc13fd6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:31.512 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.534+0000 7fcc13fd6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:31.512 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.534+0000 7fcc13fd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:17:31.512 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.534+0000 7fcc13fd6640 1 -- 172.21.15.47:0/3937464133 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcc0c075f10 con 0x7fcc0c13f650 2024-03-29T11:17:31.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.535+0000 7fcc027fc640 1 -- 172.21.15.47:0/3937464133 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7fcc08051100 con 0x7fcc0c13f650 2024-03-29T11:17:31.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.536+0000 7fcc027fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:31.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.536+0000 7fcc027fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:17:31.515 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:17:31.516 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:17:31.517 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.539+0000 7fcc13fd6640 10 librados: watch_flush enter 2024-03-29T11:17:31.517 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.539+0000 7fcc13fd6640 10 librados: watch_flush exit 2024-03-29T11:17:31.517 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.539+0000 7fcc13fd6640 20 client.4390.objecter shutdown clearing up homeless session... 2024-03-29T11:17:31.517 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.539+0000 7fcc13fd6640 10 client.4390.objecter successfully canceled tick 2024-03-29T11:17:31.517 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.539+0000 7fcc13fd6640 1 -- 172.21.15.47:0/3937464133 >> v1:172.21.15.121:6816/1964117370 conn(0x7fcbf006dcb0 legacy=0x7fcbf0070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:31.517 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.539+0000 7fcc13fd6640 10 monclient: shutdown 2024-03-29T11:17:31.517 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.539+0000 7fcc13fd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:31.517 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.539+0000 7fcc13fd6640 1 -- 172.21.15.47:0/3937464133 >> v1:172.21.15.121:6789/0 conn(0x7fcc0c13f650 legacy=0x7fcc0c140260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:31.517 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.540+0000 7fcc13fd6640 1 -- 172.21.15.47:0/3937464133 shutdown_connections 2024-03-29T11:17:31.517 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.540+0000 7fcc13fd6640 1 -- 172.21.15.47:0/3937464133 >> 172.21.15.47:0/3937464133 conn(0x7fcc0c066a00 msgr2=0x7fcc0c0b2520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:31.518 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.540+0000 7fcc13fd6640 1 -- 172.21.15.47:0/3937464133 shutdown_connections 2024-03-29T11:17:31.518 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.540+0000 7fcc13fd6640 1 -- 172.21.15.47:0/3937464133 wait complete. 2024-03-29T11:17:31.518 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:31.540+0000 7fcc13fd6640 1 librados: shutdown 2024-03-29T11:17:31.536 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:17:38.344 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:17:38.493 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.515+0000 7f7bcb58f640 10 monclient: get_monmap_and_config 2024-03-29T11:17:38.493 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.515+0000 7f7bcb58f640 10 monclient: build_initial_monmap 2024-03-29T11:17:38.493 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.515+0000 7f7bcb58f640 10 monclient: monmap: 2024-03-29T11:17:38.493 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:17:38.493 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:17:38.493 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:17:38.516044+0000 2024-03-29T11:17:38.493 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:17:38.516044+0000 2024-03-29T11:17:38.493 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:38.493 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:17:38.493 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:17:38.494 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:17:38.494 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:17:38.494 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:17:38.495 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.516+0000 7f7bcb58f640 1 Processor -- start 2024-03-29T11:17:38.495 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.516+0000 7f7bcb58f640 1 -- start start 2024-03-29T11:17:38.496 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.516+0000 7f7bcb58f640 10 monclient: init 2024-03-29T11:17:38.496 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.517+0000 7f7bcb58f640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:38.496 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.517+0000 7f7bcb58f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:17:38.496 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.517+0000 7f7bcb58f640 10 monclient(hunting): picked mon.noname-c con 0x7f7bc4070170 addr v1:172.21.15.121:6789/0 2024-03-29T11:17:38.496 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.518+0000 7f7bcb58f640 10 monclient(hunting): picked mon.noname-b con 0x7f7bc40713f0 addr v1:172.21.15.47:6790/0 2024-03-29T11:17:38.496 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.518+0000 7f7bcb58f640 10 monclient(hunting): picked mon.noname-a con 0x7f7bc4070550 addr v1:172.21.15.47:6789/0 2024-03-29T11:17:38.496 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.518+0000 7f7bcb58f640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bc413dbd0 con 0x7f7bc4070550 2024-03-29T11:17:38.496 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.518+0000 7f7bcb58f640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bc413ff70 con 0x7f7bc4070170 2024-03-29T11:17:38.496 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.518+0000 7f7bcb58f640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bc4140250 con 0x7f7bc40713f0 2024-03-29T11:17:38.496 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.518+0000 7f7bcb58f640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:38.496 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.518+0000 7f7bcb58f640 10 monclient(hunting): authenticate will time out at 1101.706035s 2024-03-29T11:17:38.496 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.518+0000 7f7bc8b2a640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f7bc40713f0 0x7f7bc41406e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:60072/0 (socket says 172.21.15.47:60072) 2024-03-29T11:17:38.496 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.518+0000 7f7bc8b2a640 1 -- 172.21.15.47:0/1155705338 learned_addr learned my addr 172.21.15.47:0/1155705338 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:17:38.496 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.518+0000 7f7bc3fff640 1 -- 172.21.15.47:0/1155705338 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 559919239 0 0) 0x7f7bc4140250 con 0x7f7bc40713f0 2024-03-29T11:17:38.497 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.518+0000 7f7bc3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:17:38.497 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.518+0000 7f7bc3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:38.497 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.518+0000 7f7bc3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:38.497 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.518+0000 7f7bc3fff640 10 monclient(hunting): my global_id is 4424 2024-03-29T11:17:38.497 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.518+0000 7f7bc3fff640 1 -- 172.21.15.47:0/1155705338 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bac003520 con 0x7f7bc40713f0 2024-03-29T11:17:38.497 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.518+0000 7f7bc3fff640 1 -- 172.21.15.47:0/1155705338 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2480445319 0 0) 0x7f7bac003520 con 0x7f7bc40713f0 2024-03-29T11:17:38.497 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.518+0000 7f7bc3fff640 1 -- 172.21.15.47:0/1155705338 >> v1:172.21.15.121:6789/0 conn(0x7f7bc4070170 legacy=0x7f7bc413d4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:17:38.498 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.521+0000 7f7bc3fff640 1 -- 172.21.15.47:0/1155705338 >> v1:172.21.15.47:6789/0 conn(0x7f7bc4070550 legacy=0x7f7bc414b7d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:17:38.501 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.523+0000 7f7bc3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:17:38.501 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.523+0000 7f7bc3fff640 1 monclient: found mon.noname-b 2024-03-29T11:17:38.501 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.523+0000 7f7bc3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:38.501 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.523+0000 7f7bc3fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-29T11:17:38.501 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.523+0000 7f7bc3fff640 1 -- 172.21.15.47:0/1155705338 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bc4140df0 con 0x7f7bc40713f0 2024-03-29T11:17:38.502 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.523+0000 7f7bc3fff640 10 monclient: _finish_auth 0 2024-03-29T11:17:38.502 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.523+0000 7f7bc3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:17:38.502 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.523+0000 7f7bc3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:38.502 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.523+0000 7f7bc3fff640 1 -- 172.21.15.47:0/1155705338 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f7bb8001690 con 0x7f7bc40713f0 2024-03-29T11:17:38.502 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.523+0000 7f7bc3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:38.502 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.523+0000 7f7bc3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:17:38.502 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.523+0000 7f7bc3fff640 10 monclient: dump: 2024-03-29T11:17:38.502 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:17:38.502 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:17:38.502 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:38.502 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:38.502 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:38.502 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:17:38.503 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:17:38.503 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:17:38.503 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:17:38.503 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:17:38.503 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.523+0000 7f7bcb58f640 5 monclient: authenticate success, global_id 4424 2024-03-29T11:17:38.503 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.523+0000 7f7bcb58f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:17:38.503 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.523+0000 7f7bc3fff640 1 -- 172.21.15.47:0/1155705338 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7bb8001860 con 0x7f7bc40713f0 2024-03-29T11:17:38.503 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.523+0000 7f7bc3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:38.503 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.523+0000 7f7bc3fff640 1 -- 172.21.15.47:0/1155705338 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f7bb8001cb0 con 0x7f7bc40713f0 2024-03-29T11:17:38.503 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.523+0000 7f7bc3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:38.503 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.523+0000 7f7bc3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:17:38.503 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.523+0000 7f7bc3fff640 10 monclient: dump: 2024-03-29T11:17:38.503 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:17:38.503 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:17:38.503 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:38.504 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:38.504 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:38.504 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:17:38.504 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:17:38.504 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:17:38.504 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:17:38.504 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:17:38.504 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.524+0000 7f7bcb58f640 10 monclient: get_monmap_and_config success 2024-03-29T11:17:38.504 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.524+0000 7f7bcb58f640 4 set_mon_vals no callback set 2024-03-29T11:17:38.505 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.527+0000 7f7bcb58f640 10 monclient: shutdown 2024-03-29T11:17:38.505 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.527+0000 7f7bcb58f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:38.505 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.527+0000 7f7bcb58f640 1 -- 172.21.15.47:0/1155705338 >> v1:172.21.15.47:6790/0 conn(0x7f7bc40713f0 legacy=0x7f7bc41406e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:38.505 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.527+0000 7f7bcb58f640 1 -- 172.21.15.47:0/1155705338 shutdown_connections 2024-03-29T11:17:38.505 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.527+0000 7f7bcb58f640 1 -- 172.21.15.47:0/1155705338 >> 172.21.15.47:0/1155705338 conn(0x7f7bc4067610 msgr2=0x7f7bc40b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:38.505 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.528+0000 7f7bcb58f640 1 -- 172.21.15.47:0/1155705338 shutdown_connections 2024-03-29T11:17:38.506 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.528+0000 7f7bcb58f640 1 -- 172.21.15.47:0/1155705338 wait complete. 2024-03-29T11:17:38.507 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.530+0000 7f7bcb58f640 10 monclient: build_initial_monmap 2024-03-29T11:17:38.507 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.530+0000 7f7bcb58f640 10 monclient: monmap: 2024-03-29T11:17:38.507 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:17:38.507 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:17:38.507 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:17:38.507 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:17:38.507 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:38.507 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:17:38.507 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:17:38.507 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:17:38.507 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:17:38.508 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:17:38.508 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.530+0000 7f7bcb58f640 1 librados: starting msgr at 2024-03-29T11:17:38.509 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.530+0000 7f7bcb58f640 1 librados: starting objecter 2024-03-29T11:17:38.509 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.530+0000 7f7bcb58f640 1 Processor -- start 2024-03-29T11:17:38.509 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.531+0000 7f7bcb58f640 1 -- start start 2024-03-29T11:17:38.509 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.531+0000 7f7bcb58f640 1 librados: setting wanted keys 2024-03-29T11:17:38.509 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.532+0000 7f7bcb58f640 1 librados: calling monclient init 2024-03-29T11:17:38.509 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.532+0000 7f7bcb58f640 10 monclient: init 2024-03-29T11:17:38.510 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.533+0000 7f7bcb58f640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:38.510 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.533+0000 7f7bcb58f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:17:38.512 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.534+0000 7f7bcb58f640 10 monclient(hunting): picked mon.noname-b con 0x7f7bc4070170 addr v1:172.21.15.121:6789/0 2024-03-29T11:17:38.512 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.534+0000 7f7bcb58f640 10 monclient(hunting): picked mon.noname-a con 0x7f7bc4070550 addr v1:172.21.15.47:6789/0 2024-03-29T11:17:38.512 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.534+0000 7f7bcb58f640 10 monclient(hunting): picked mon.noname-c con 0x7f7bc40713f0 addr v1:172.21.15.47:6790/0 2024-03-29T11:17:38.512 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.534+0000 7f7bcb58f640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bc41391a0 con 0x7f7bc4070550 2024-03-29T11:17:38.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.534+0000 7f7bcb58f640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bc413a6c0 con 0x7f7bc4070170 2024-03-29T11:17:38.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.534+0000 7f7bcb58f640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bc413a9a0 con 0x7f7bc40713f0 2024-03-29T11:17:38.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.534+0000 7f7bcb58f640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:38.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.534+0000 7f7bcb58f640 10 monclient(hunting): authenticate will time out at 1101.722165s 2024-03-29T11:17:38.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.534+0000 7f7bc9b2c640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f7bc40713f0 0x7f7bc41e5d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:60958/0 (socket says 172.21.15.47:60958) 2024-03-29T11:17:38.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.535+0000 7f7bc9b2c640 1 -- 172.21.15.47:0/4271675132 learned_addr learned my addr 172.21.15.47:0/4271675132 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:17:38.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.535+0000 7f7bc1ffb640 10 client.?.objecter ms_handle_connect 0x7f7bc40713f0 2024-03-29T11:17:38.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.535+0000 7f7bc1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:38.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.535+0000 7f7bc1ffb640 1 -- 172.21.15.47:0/4271675132 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3467230098 0 0) 0x7f7bc413a9a0 con 0x7f7bc40713f0 2024-03-29T11:17:38.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.535+0000 7f7bc1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:17:38.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.535+0000 7f7bc1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:38.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.535+0000 7f7bc1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:38.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.535+0000 7f7bc1ffb640 10 monclient(hunting): my global_id is 4427 2024-03-29T11:17:38.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.535+0000 7f7bc1ffb640 1 -- 172.21.15.47:0/4271675132 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bb0004eb0 con 0x7f7bc40713f0 2024-03-29T11:17:38.515 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.536+0000 7f7bc1ffb640 1 -- 172.21.15.47:0/4271675132 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1867550500 0 0) 0x7f7bb0004eb0 con 0x7f7bc40713f0 2024-03-29T11:17:38.516 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.536+0000 7f7bc1ffb640 1 -- 172.21.15.47:0/4271675132 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7bc413a9a0 con 0x7f7bc40713f0 2024-03-29T11:17:38.516 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.536+0000 7f7bc1ffb640 1 -- 172.21.15.47:0/4271675132 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f7bb4001c60 con 0x7f7bc40713f0 2024-03-29T11:17:38.516 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.537+0000 7f7bc1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:38.516 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.537+0000 7f7bc1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:17:38.516 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.537+0000 7f7bc1ffb640 10 monclient(hunting): dump: 2024-03-29T11:17:38.516 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:17:38.516 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:17:38.516 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:38.516 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:38.516 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:38.516 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:17:38.516 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:17:38.516 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:17:38.517 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:17:38.517 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:17:38.517 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.537+0000 7f7bc1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:17:38.517 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.537+0000 7f7bc1ffb640 1 -- 172.21.15.47:0/4271675132 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3826556671 0 0) 0x7f7bc413a9a0 con 0x7f7bc40713f0 2024-03-29T11:17:38.517 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.537+0000 7f7bc1ffb640 1 -- 172.21.15.47:0/4271675132 >> v1:172.21.15.121:6789/0 conn(0x7f7bc4070170 legacy=0x7f7bc4138900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:38.517 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.537+0000 7f7bc1ffb640 1 -- 172.21.15.47:0/4271675132 >> v1:172.21.15.47:6789/0 conn(0x7f7bc4070550 legacy=0x7f7bc4139630 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:17:38.517 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.538+0000 7f7bc1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:17:38.517 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.538+0000 7f7bc1ffb640 1 monclient: found mon.c 2024-03-29T11:17:38.517 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.538+0000 7f7bc1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:38.517 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.538+0000 7f7bc1ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:17:38.517 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.538+0000 7f7bc1ffb640 1 -- 172.21.15.47:0/4271675132 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bc4139d40 con 0x7f7bc40713f0 2024-03-29T11:17:38.517 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.538+0000 7f7bc1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:17:38.517 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.538+0000 7f7bc1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:17:38.517 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.538+0000 7f7bc1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:38.517 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.538+0000 7f7bcb58f640 5 monclient: authenticate success, global_id 4427 2024-03-29T11:17:38.517 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.538+0000 7f7bcb58f640 10 monclient: _renew_subs 2024-03-29T11:17:38.518 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.538+0000 7f7bcb58f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:17:38.518 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.538+0000 7f7bcb58f640 1 -- 172.21.15.47:0/4271675132 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bc41de7b0 con 0x7f7bc40713f0 2024-03-29T11:17:38.520 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.539+0000 7f7bcb58f640 10 client.4427.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:17:38.520 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.539+0000 7f7bcb58f640 10 monclient: _renew_subs 2024-03-29T11:17:38.520 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.539+0000 7f7bcb58f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:17:38.520 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.539+0000 7f7bcb58f640 1 -- 172.21.15.47:0/4271675132 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bc413cb60 con 0x7f7bc40713f0 2024-03-29T11:17:38.520 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.539+0000 7f7bcb58f640 1 librados: init done 2024-03-29T11:17:38.520 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.540+0000 7f7bc1ffb640 1 -- 172.21.15.47:0/4271675132 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7bb40020b0 con 0x7f7bc40713f0 2024-03-29T11:17:38.520 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.540+0000 7f7bc1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:38.520 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.540+0000 7f7bc1ffb640 1 -- 172.21.15.47:0/4271675132 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f7bb4003c20 con 0x7f7bc40713f0 2024-03-29T11:17:38.520 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.540+0000 7f7bc1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:38.520 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.540+0000 7f7bc1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:17:38.520 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.540+0000 7f7bc1ffb640 10 monclient: dump: 2024-03-29T11:17:38.521 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:17:38.521 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:17:38.521 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:38.521 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:38.521 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:38.521 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:17:38.521 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:17:38.521 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:17:38.521 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:17:38.521 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:17:38.521 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.540+0000 7f7bc37fe640 4 set_mon_vals no callback set 2024-03-29T11:17:38.521 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.541+0000 7f7bcb58f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:38.522 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.543+0000 7f7bcb58f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:38.522 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.543+0000 7f7bcb58f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:17:38.522 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.543+0000 7f7bcb58f640 1 -- 172.21.15.47:0/4271675132 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bc4075f10 con 0x7f7bc40713f0 2024-03-29T11:17:38.522 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.544+0000 7f7bc1ffb640 1 -- 172.21.15.47:0/4271675132 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7f7bb4003d80 con 0x7f7bc40713f0 2024-03-29T11:17:38.523 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.545+0000 7f7bc1ffb640 1 -- 172.21.15.47:0/4271675132 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7f7bb4084df0 con 0x7f7bc40713f0 2024-03-29T11:17:38.523 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.546+0000 7f7bc1ffb640 10 client.4427.objecter ms_dispatch 0x7f7bc4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:17:38.523 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.546+0000 7f7bc932b640 10 monclient: get_auth_request con 0x7f7bb006dbc0 auth_method 0 2024-03-29T11:17:38.523 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.546+0000 7f7bc1ffb640 3 client.4427.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:17:38.523 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.546+0000 7f7bc1ffb640 3 client.4427.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:17:38.523 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.546+0000 7f7bc1ffb640 20 client.4427.objecter dump_active .. 0 homeless 2024-03-29T11:17:38.523 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.546+0000 7f7bc1ffb640 10 client.4427.objecter ms_handle_connect 0x7f7bb006dbc0 2024-03-29T11:17:38.532 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.554+0000 7f7bc1ffb640 1 -- 172.21.15.47:0/4271675132 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7bb40506d0 con 0x7f7bc40713f0 2024-03-29T11:17:38.532 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.554+0000 7f7bc1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:38.532 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.554+0000 7f7bc1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:17:38.876 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.899+0000 7f7bcb58f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:38.877 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.899+0000 7f7bcb58f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:38.877 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.899+0000 7f7bcb58f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:17:38.877 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.899+0000 7f7bcb58f640 1 -- 172.21.15.47:0/4271675132 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7bc4074f50 con 0x7f7bc40713f0 2024-03-29T11:17:38.877 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.900+0000 7f7bc1ffb640 1 -- 172.21.15.47:0/4271675132 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7f7bb4052600 con 0x7f7bc40713f0 2024-03-29T11:17:38.878 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.900+0000 7f7bc1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:38.878 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.900+0000 7f7bc1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:17:38.878 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:17:38.878 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:17:38.884 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.906+0000 7f7bcb58f640 10 librados: watch_flush enter 2024-03-29T11:17:38.884 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.906+0000 7f7bcb58f640 10 librados: watch_flush exit 2024-03-29T11:17:38.884 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.906+0000 7f7bcb58f640 20 client.4427.objecter shutdown clearing up homeless session... 2024-03-29T11:17:38.884 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.906+0000 7f7bcb58f640 10 client.4427.objecter successfully canceled tick 2024-03-29T11:17:38.884 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.906+0000 7f7bcb58f640 1 -- 172.21.15.47:0/4271675132 >> v1:172.21.15.121:6816/1964117370 conn(0x7f7bb006dbc0 legacy=0x7f7bb0070080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:38.884 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.906+0000 7f7bcb58f640 10 monclient: shutdown 2024-03-29T11:17:38.884 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.906+0000 7f7bcb58f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:38.884 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.906+0000 7f7bcb58f640 1 -- 172.21.15.47:0/4271675132 >> v1:172.21.15.47:6790/0 conn(0x7f7bc40713f0 legacy=0x7f7bc41e5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:38.884 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.906+0000 7f7bcb58f640 1 -- 172.21.15.47:0/4271675132 shutdown_connections 2024-03-29T11:17:38.884 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.906+0000 7f7bcb58f640 1 -- 172.21.15.47:0/4271675132 >> 172.21.15.47:0/4271675132 conn(0x7f7bc4067610 msgr2=0x7f7bc4069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:38.884 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.907+0000 7f7bcb58f640 1 -- 172.21.15.47:0/4271675132 shutdown_connections 2024-03-29T11:17:38.885 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.907+0000 7f7bcb58f640 1 -- 172.21.15.47:0/4271675132 wait complete. 2024-03-29T11:17:38.885 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:38.908+0000 7f7bcb58f640 1 librados: shutdown 2024-03-29T11:17:38.903 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:17:45.709 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:17:45.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.882+0000 7fc4a6d0b640 10 monclient: get_monmap_and_config 2024-03-29T11:17:45.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.882+0000 7fc4a6d0b640 10 monclient: build_initial_monmap 2024-03-29T11:17:45.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.882+0000 7fc4a6d0b640 10 monclient: monmap: 2024-03-29T11:17:45.860 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:17:45.860 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:17:45.860 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:17:45.883058+0000 2024-03-29T11:17:45.860 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:17:45.883058+0000 2024-03-29T11:17:45.861 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:45.861 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:17:45.861 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:17:45.861 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:17:45.861 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:17:45.861 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:17:45.861 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.883+0000 7fc4a6d0b640 1 Processor -- start 2024-03-29T11:17:45.861 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.883+0000 7fc4a6d0b640 1 -- start start 2024-03-29T11:17:45.861 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.883+0000 7fc4a6d0b640 10 monclient: init 2024-03-29T11:17:45.866 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.884+0000 7fc4a6d0b640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:45.866 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.884+0000 7fc4a6d0b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:17:45.866 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.884+0000 7fc4a6d0b640 10 monclient(hunting): picked mon.noname-a con 0x7fc4a00ae220 addr v1:172.21.15.47:6789/0 2024-03-29T11:17:45.866 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.884+0000 7fc4a6d0b640 10 monclient(hunting): picked mon.noname-c con 0x7fc4a0072f10 addr v1:172.21.15.121:6789/0 2024-03-29T11:17:45.866 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.884+0000 7fc4a6d0b640 10 monclient(hunting): picked mon.noname-b con 0x7fc4a00acea0 addr v1:172.21.15.47:6790/0 2024-03-29T11:17:45.866 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.884+0000 7fc4a6d0b640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4a00aa9f0 con 0x7fc4a00ae220 2024-03-29T11:17:45.866 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.884+0000 7fc4a6d0b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4a00ac7a0 con 0x7fc4a0072f10 2024-03-29T11:17:45.866 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.884+0000 7fc4a6d0b640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4a00ab5a0 con 0x7fc4a00acea0 2024-03-29T11:17:45.866 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.884+0000 7fc4a6d0b640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:45.867 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.884+0000 7fc4a6d0b640 10 monclient(hunting): authenticate will time out at 1109.072318s 2024-03-29T11:17:45.867 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.885+0000 7fc4a650a640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fc4a00acea0 0x7fc4a00ad2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:40774/0 (socket says 172.21.15.47:40774) 2024-03-29T11:17:45.867 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.885+0000 7fc4a650a640 1 -- 172.21.15.47:0/3464239923 learned_addr learned my addr 172.21.15.47:0/3464239923 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:17:45.867 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.886+0000 7fc4a4d07640 1 -- 172.21.15.47:0/3464239923 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 146091444 0 0) 0x7fc4a00ab5a0 con 0x7fc4a00acea0 2024-03-29T11:17:45.867 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.886+0000 7fc4a4d07640 10 monclient(hunting): _negotiate 2024-03-29T11:17:45.867 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.886+0000 7fc4a4d07640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:45.867 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.886+0000 7fc4a4d07640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:45.867 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.886+0000 7fc4a4d07640 10 monclient(hunting): my global_id is 4430 2024-03-29T11:17:45.867 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.886+0000 7fc4a4d07640 1 -- 172.21.15.47:0/3464239923 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc48c00bd30 con 0x7fc4a00acea0 2024-03-29T11:17:45.867 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.887+0000 7fc4a4d07640 1 -- 172.21.15.47:0/3464239923 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3330248880 0 0) 0x7fc48c00bd30 con 0x7fc4a00acea0 2024-03-29T11:17:45.867 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.887+0000 7fc4a4d07640 1 -- 172.21.15.47:0/3464239923 >> v1:172.21.15.121:6789/0 conn(0x7fc4a0072f10 legacy=0x7fc4a0141d60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:17:45.867 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.887+0000 7fc4a4d07640 1 -- 172.21.15.47:0/3464239923 >> v1:172.21.15.47:6789/0 conn(0x7fc4a00ae220 legacy=0x7fc4a0141450 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:17:45.868 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.890+0000 7fc4a4d07640 10 monclient: _finish_hunting 0 2024-03-29T11:17:45.868 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.890+0000 7fc4a4d07640 1 monclient: found mon.noname-b 2024-03-29T11:17:45.868 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.890+0000 7fc4a4d07640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:45.868 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.890+0000 7fc4a4d07640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-29T11:17:45.868 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.890+0000 7fc4a4d07640 1 -- 172.21.15.47:0/3464239923 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4a00ab730 con 0x7fc4a00acea0 2024-03-29T11:17:45.868 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.890+0000 7fc4a4d07640 10 monclient: _finish_auth 0 2024-03-29T11:17:45.868 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.890+0000 7fc4a4d07640 10 monclient: _check_auth_tickets 2024-03-29T11:17:45.868 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.890+0000 7fc4a4d07640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:45.868 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.890+0000 7fc4a4d07640 1 -- 172.21.15.47:0/3464239923 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fc490001690 con 0x7fc4a00acea0 2024-03-29T11:17:45.869 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.890+0000 7fc4a4d07640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:45.869 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.890+0000 7fc4a4d07640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:17:45.869 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.890+0000 7fc4a4d07640 10 monclient: dump: 2024-03-29T11:17:45.869 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:17:45.869 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:17:45.869 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:45.869 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:45.869 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:45.869 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:17:45.869 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:17:45.869 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:17:45.869 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:17:45.869 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:17:45.869 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.891+0000 7fc4a6d0b640 5 monclient: authenticate success, global_id 4430 2024-03-29T11:17:45.870 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.891+0000 7fc4a6d0b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:17:45.870 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.893+0000 7fc4a4d07640 1 -- 172.21.15.47:0/3464239923 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc4900017f0 con 0x7fc4a00acea0 2024-03-29T11:17:45.870 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.893+0000 7fc4a4d07640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:45.871 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.893+0000 7fc4a6d0b640 10 monclient: get_monmap_and_config success 2024-03-29T11:17:45.871 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.893+0000 7fc4a6d0b640 4 set_mon_vals no callback set 2024-03-29T11:17:45.871 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.894+0000 7fc4a6d0b640 10 monclient: shutdown 2024-03-29T11:17:45.871 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.894+0000 7fc4a6d0b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:45.872 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.894+0000 7fc4a6d0b640 1 -- 172.21.15.47:0/3464239923 >> v1:172.21.15.47:6790/0 conn(0x7fc4a00acea0 legacy=0x7fc4a00ad2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:45.873 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.895+0000 7fc4a6d0b640 1 -- 172.21.15.47:0/3464239923 shutdown_connections 2024-03-29T11:17:45.873 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.895+0000 7fc4a6d0b640 1 -- 172.21.15.47:0/3464239923 >> 172.21.15.47:0/3464239923 conn(0x7fc4a00b3a30 msgr2=0x7fc4a00b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:45.874 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.897+0000 7fc4a6d0b640 1 -- 172.21.15.47:0/3464239923 shutdown_connections 2024-03-29T11:17:45.874 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.897+0000 7fc4a6d0b640 1 -- 172.21.15.47:0/3464239923 wait complete. 2024-03-29T11:17:45.875 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.898+0000 7fc4a6d0b640 10 monclient: build_initial_monmap 2024-03-29T11:17:45.876 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.898+0000 7fc4a6d0b640 10 monclient: monmap: 2024-03-29T11:17:45.876 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:17:45.876 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:17:45.876 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:17:45.876 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:17:45.876 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:45.876 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:17:45.876 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:17:45.876 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:17:45.876 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:17:45.876 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:17:45.876 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.898+0000 7fc4a6d0b640 1 librados: starting msgr at 2024-03-29T11:17:45.876 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.898+0000 7fc4a6d0b640 1 librados: starting objecter 2024-03-29T11:17:45.876 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.898+0000 7fc4a6d0b640 1 Processor -- start 2024-03-29T11:17:45.876 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.898+0000 7fc4a6d0b640 1 -- start start 2024-03-29T11:17:45.877 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.898+0000 7fc4a6d0b640 1 librados: setting wanted keys 2024-03-29T11:17:45.877 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.898+0000 7fc4a6d0b640 1 librados: calling monclient init 2024-03-29T11:17:45.877 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.898+0000 7fc4a6d0b640 10 monclient: init 2024-03-29T11:17:45.878 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.899+0000 7fc4a6d0b640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:45.878 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.899+0000 7fc4a6d0b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:17:45.878 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.899+0000 7fc4a6d0b640 10 monclient(hunting): picked mon.noname-c con 0x7fc4a0072f10 addr v1:172.21.15.47:6790/0 2024-03-29T11:17:45.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.899+0000 7fc4a6d0b640 10 monclient(hunting): picked mon.noname-a con 0x7fc4a00acea0 addr v1:172.21.15.47:6789/0 2024-03-29T11:17:45.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.900+0000 7fc4a6d0b640 10 monclient(hunting): picked mon.noname-b con 0x7fc4a00ae220 addr v1:172.21.15.121:6789/0 2024-03-29T11:17:45.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.900+0000 7fc4a6d0b640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4a013df50 con 0x7fc4a00acea0 2024-03-29T11:17:45.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.900+0000 7fc4a6d0b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4a013e230 con 0x7fc4a00ae220 2024-03-29T11:17:45.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.900+0000 7fc4a6d0b640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4a013c640 con 0x7fc4a0072f10 2024-03-29T11:17:45.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.900+0000 7fc4a6d0b640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:45.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.900+0000 7fc4a6d0b640 10 monclient(hunting): authenticate will time out at 1109.088426s 2024-03-29T11:17:45.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.901+0000 7fc4a5d09640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fc4a0072f10 0x7fc4a013f250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:42394/0 (socket says 172.21.15.47:42394) 2024-03-29T11:17:45.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.901+0000 7fc4a5d09640 1 -- 172.21.15.47:0/1290288902 learned_addr learned my addr 172.21.15.47:0/1290288902 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:17:45.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.901+0000 7fc4967fc640 10 client.?.objecter ms_handle_connect 0x7fc4a0072f10 2024-03-29T11:17:45.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.901+0000 7fc4967fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:45.883 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.904+0000 7fc4967fc640 10 client.?.objecter ms_handle_connect 0x7fc4a00ae220 2024-03-29T11:17:45.884 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.904+0000 7fc4967fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:45.884 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.904+0000 7fc4967fc640 1 -- 172.21.15.47:0/1290288902 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 655116222 0 0) 0x7fc4a013c640 con 0x7fc4a0072f10 2024-03-29T11:17:45.884 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.904+0000 7fc4967fc640 10 monclient(hunting): _negotiate 2024-03-29T11:17:45.884 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.904+0000 7fc4967fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:45.884 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.904+0000 7fc4967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:45.884 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.904+0000 7fc4967fc640 10 monclient(hunting): my global_id is 4433 2024-03-29T11:17:45.884 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.904+0000 7fc4967fc640 1 -- 172.21.15.47:0/1290288902 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc484004eb0 con 0x7fc4a0072f10 2024-03-29T11:17:45.885 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.904+0000 7fc4967fc640 1 -- 172.21.15.47:0/1290288902 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 88953431 0 0) 0x7fc4a013e230 con 0x7fc4a00ae220 2024-03-29T11:17:45.885 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.904+0000 7fc4967fc640 10 monclient(hunting): _negotiate 2024-03-29T11:17:45.885 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.904+0000 7fc4967fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:45.885 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.904+0000 7fc4967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:45.885 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.904+0000 7fc4967fc640 10 monclient(hunting): my global_id is 4393 2024-03-29T11:17:45.885 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.904+0000 7fc4967fc640 1 -- 172.21.15.47:0/1290288902 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4840051d0 con 0x7fc4a00ae220 2024-03-29T11:17:45.885 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.905+0000 7fc4967fc640 1 -- 172.21.15.47:0/1290288902 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 547711742 0 0) 0x7fc4840051d0 con 0x7fc4a00ae220 2024-03-29T11:17:45.885 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.905+0000 7fc4967fc640 1 -- 172.21.15.47:0/1290288902 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc484005640 con 0x7fc4a00ae220 2024-03-29T11:17:45.885 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.905+0000 7fc4967fc640 1 -- 172.21.15.47:0/1290288902 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2052030762 0 0) 0x7fc484004eb0 con 0x7fc4a0072f10 2024-03-29T11:17:45.885 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.905+0000 7fc4967fc640 1 -- 172.21.15.47:0/1290288902 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc4840051d0 con 0x7fc4a0072f10 2024-03-29T11:17:45.885 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.905+0000 7fc4967fc640 1 -- 172.21.15.47:0/1290288902 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fc490000fe0 con 0x7fc4a00ae220 2024-03-29T11:17:45.885 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.905+0000 7fc4967fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:45.885 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.905+0000 7fc4967fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:17:45.886 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.905+0000 7fc4967fc640 10 monclient(hunting): dump: 2024-03-29T11:17:45.886 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:17:45.886 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:17:45.886 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:45.886 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:45.886 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:45.886 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:17:45.886 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:17:45.886 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:17:45.886 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:17:45.886 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:17:45.886 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.905+0000 7fc4967fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:17:45.886 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.905+0000 7fc4967fc640 1 -- 172.21.15.47:0/1290288902 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fc498001c20 con 0x7fc4a0072f10 2024-03-29T11:17:45.886 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.905+0000 7fc4967fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:45.886 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.905+0000 7fc4967fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:17:45.887 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.905+0000 7fc4967fc640 10 monclient(hunting): dump: 2024-03-29T11:17:45.887 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:17:45.887 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:17:45.887 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:45.887 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:45.887 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:45.887 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:17:45.887 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:17:45.887 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:17:45.887 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:17:45.887 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:17:45.887 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.905+0000 7fc4967fc640 1 -- 172.21.15.47:0/1290288902 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2761766390 0 0) 0x7fc484005640 con 0x7fc4a00ae220 2024-03-29T11:17:45.887 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.905+0000 7fc4967fc640 1 -- 172.21.15.47:0/1290288902 >> v1:172.21.15.47:6790/0 conn(0x7fc4a0072f10 legacy=0x7fc4a013f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:45.887 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.905+0000 7fc4967fc640 1 -- 172.21.15.47:0/1290288902 >> v1:172.21.15.47:6789/0 conn(0x7fc4a00acea0 legacy=0x7fc4a013fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:45.887 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.905+0000 7fc4967fc640 10 monclient: _finish_hunting 0 2024-03-29T11:17:45.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.905+0000 7fc4967fc640 1 monclient: found mon.b 2024-03-29T11:17:45.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.905+0000 7fc4967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:45.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.905+0000 7fc4967fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:17:45.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.905+0000 7fc4967fc640 1 -- 172.21.15.47:0/1290288902 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4a0141810 con 0x7fc4a00ae220 2024-03-29T11:17:45.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.905+0000 7fc4967fc640 10 monclient: _finish_auth 0 2024-03-29T11:17:45.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.905+0000 7fc4967fc640 10 monclient: _check_auth_tickets 2024-03-29T11:17:45.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.905+0000 7fc4967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:45.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.905+0000 7fc4a6d0b640 5 monclient: authenticate success, global_id 4393 2024-03-29T11:17:45.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.905+0000 7fc4a6d0b640 10 monclient: _renew_subs 2024-03-29T11:17:45.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.905+0000 7fc4a6d0b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:17:45.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.905+0000 7fc4a6d0b640 1 -- 172.21.15.47:0/1290288902 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4a013c230 con 0x7fc4a00ae220 2024-03-29T11:17:45.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.906+0000 7fc4a6d0b640 10 client.4393.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:17:45.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.906+0000 7fc4a6d0b640 10 monclient: _renew_subs 2024-03-29T11:17:45.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.906+0000 7fc4a6d0b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:17:45.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.906+0000 7fc4a6d0b640 1 -- 172.21.15.47:0/1290288902 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4a013eba0 con 0x7fc4a00ae220 2024-03-29T11:17:45.889 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.906+0000 7fc4a6d0b640 1 librados: init done 2024-03-29T11:17:45.889 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.907+0000 7fc4967fc640 1 -- 172.21.15.47:0/1290288902 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc490004bf0 con 0x7fc4a00ae220 2024-03-29T11:17:45.889 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.907+0000 7fc4967fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:45.889 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.907+0000 7fc4967fc640 1 -- 172.21.15.47:0/1290288902 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fc490004f80 con 0x7fc4a00ae220 2024-03-29T11:17:45.889 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.907+0000 7fc4967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:45.889 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.907+0000 7fc4967fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:17:45.889 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.907+0000 7fc4967fc640 10 monclient: dump: 2024-03-29T11:17:45.889 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:17:45.889 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:17:45.889 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:45.889 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:45.889 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:45.889 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:17:45.889 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:17:45.890 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:17:45.890 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:17:45.890 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:17:45.890 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.907+0000 7fc497fff640 4 set_mon_vals no callback set 2024-03-29T11:17:45.891 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.908+0000 7fc48bfff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:45.891 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.908+0000 7fc48bfff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:45.891 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.908+0000 7fc48bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:17:45.891 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.908+0000 7fc48bfff640 1 -- 172.21.15.47:0/1290288902 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4a004f730 con 0x7fc4a00ae220 2024-03-29T11:17:45.891 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.909+0000 7fc4967fc640 1 -- 172.21.15.47:0/1290288902 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7fc490001aa0 con 0x7fc4a00ae220 2024-03-29T11:17:45.891 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.909+0000 7fc4967fc640 1 -- 172.21.15.47:0/1290288902 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7fc4900849b0 con 0x7fc4a00ae220 2024-03-29T11:17:45.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.909+0000 7fc4967fc640 10 client.4393.objecter ms_dispatch 0x7fc4a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:17:45.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.909+0000 7fc4967fc640 3 client.4393.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:17:45.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.909+0000 7fc4967fc640 3 client.4393.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:17:45.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.909+0000 7fc4967fc640 20 client.4393.objecter dump_active .. 0 homeless 2024-03-29T11:17:45.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.910+0000 7fc4a5d09640 10 monclient: get_auth_request con 0x7fc48407f240 auth_method 0 2024-03-29T11:17:45.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.910+0000 7fc4967fc640 10 client.4393.objecter ms_handle_connect 0x7fc48407f240 2024-03-29T11:17:45.897 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.920+0000 7fc4967fc640 1 -- 172.21.15.47:0/1290288902 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc4900502f0 con 0x7fc4a00ae220 2024-03-29T11:17:45.897 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.920+0000 7fc4967fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:45.897 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:45.920+0000 7fc4967fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:17:46.223 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:46.245+0000 7fc48bfff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:46.223 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:46.245+0000 7fc48bfff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:46.223 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:46.245+0000 7fc48bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:17:46.223 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:46.245+0000 7fc48bfff640 1 -- 172.21.15.47:0/1290288902 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc4a01419a0 con 0x7fc4a00ae220 2024-03-29T11:17:46.226 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:46.249+0000 7fc4967fc640 1 -- 172.21.15.47:0/1290288902 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7fc490052220 con 0x7fc4a00ae220 2024-03-29T11:17:46.226 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:46.249+0000 7fc4967fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:46.226 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:46.249+0000 7fc4967fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:17:46.228 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:17:46.228 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:17:46.229 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:46.252+0000 7fc48bfff640 10 librados: watch_flush enter 2024-03-29T11:17:46.229 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:46.252+0000 7fc48bfff640 10 librados: watch_flush exit 2024-03-29T11:17:46.229 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:46.252+0000 7fc48bfff640 20 client.4393.objecter shutdown clearing up homeless session... 2024-03-29T11:17:46.229 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:46.252+0000 7fc48bfff640 10 client.4393.objecter successfully canceled tick 2024-03-29T11:17:46.230 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:46.253+0000 7fc48bfff640 1 -- 172.21.15.47:0/1290288902 >> v1:172.21.15.121:6816/1964117370 conn(0x7fc48407f240 legacy=0x7fc484081700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:46.230 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:46.253+0000 7fc48bfff640 10 monclient: shutdown 2024-03-29T11:17:46.230 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:46.253+0000 7fc48bfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:46.231 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:46.253+0000 7fc48bfff640 1 -- 172.21.15.47:0/1290288902 >> v1:172.21.15.121:6789/0 conn(0x7fc4a00ae220 legacy=0x7fc4a01de110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:46.231 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:46.253+0000 7fc48bfff640 1 -- 172.21.15.47:0/1290288902 shutdown_connections 2024-03-29T11:17:46.231 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:46.253+0000 7fc48bfff640 1 -- 172.21.15.47:0/1290288902 >> 172.21.15.47:0/1290288902 conn(0x7fc4a00b3a30 msgr2=0x7fc4a00b3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:46.231 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:46.253+0000 7fc48bfff640 1 -- 172.21.15.47:0/1290288902 shutdown_connections 2024-03-29T11:17:46.231 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:46.253+0000 7fc48bfff640 1 -- 172.21.15.47:0/1290288902 wait complete. 2024-03-29T11:17:46.231 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:46.253+0000 7fc48bfff640 1 librados: shutdown 2024-03-29T11:17:46.250 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:17:53.058 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:17:53.208 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.227+0000 7fab67fff640 10 monclient: get_monmap_and_config 2024-03-29T11:17:53.208 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.227+0000 7fab67fff640 10 monclient: build_initial_monmap 2024-03-29T11:17:53.208 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.227+0000 7fab67fff640 10 monclient: monmap: 2024-03-29T11:17:53.208 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:17:53.208 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:17:53.208 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:17:53.228849+0000 2024-03-29T11:17:53.208 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:17:53.228849+0000 2024-03-29T11:17:53.208 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:53.208 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:17:53.209 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:17:53.209 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:17:53.209 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:17:53.209 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:17:53.209 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.230+0000 7fab67fff640 1 Processor -- start 2024-03-29T11:17:53.209 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.230+0000 7fab67fff640 1 -- start start 2024-03-29T11:17:53.209 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.230+0000 7fab67fff640 10 monclient: init 2024-03-29T11:17:53.209 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.231+0000 7fab67fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:53.209 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.231+0000 7fab67fff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:17:53.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.232+0000 7fab67fff640 10 monclient(hunting): picked mon.noname-a con 0x7fab68071f40 addr v1:172.21.15.47:6789/0 2024-03-29T11:17:53.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.232+0000 7fab67fff640 10 monclient(hunting): picked mon.noname-b con 0x7fab680adcb0 addr v1:172.21.15.47:6790/0 2024-03-29T11:17:53.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.232+0000 7fab67fff640 10 monclient(hunting): picked mon.noname-c con 0x7fab6813f910 addr v1:172.21.15.121:6789/0 2024-03-29T11:17:53.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.232+0000 7fab67fff640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab680af000 con 0x7fab68071f40 2024-03-29T11:17:53.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.232+0000 7fab67fff640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab680ab060 con 0x7fab6813f910 2024-03-29T11:17:53.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.232+0000 7fab67fff640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab680aec40 con 0x7fab680adcb0 2024-03-29T11:17:53.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.232+0000 7fab67fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:53.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.232+0000 7fab67fff640 10 monclient(hunting): authenticate will time out at 1116.420534s 2024-03-29T11:17:53.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.233+0000 7fab667fc640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fab680adcb0 0x7fab68145560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:34668/0 (socket says 172.21.15.47:34668) 2024-03-29T11:17:53.213 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.233+0000 7fab667fc640 1 -- 172.21.15.47:0/569620098 learned_addr learned my addr 172.21.15.47:0/569620098 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:17:53.218 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.238+0000 7fab65ffb640 1 -- 172.21.15.47:0/569620098 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1678891386 0 0) 0x7fab680af000 con 0x7fab68071f40 2024-03-29T11:17:53.218 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.238+0000 7fab65ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:17:53.218 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.238+0000 7fab65ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:53.218 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.238+0000 7fab65ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:53.218 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.238+0000 7fab65ffb640 10 monclient(hunting): my global_id is 4380 2024-03-29T11:17:53.219 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.238+0000 7fab65ffb640 1 -- 172.21.15.47:0/569620098 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab54003520 con 0x7fab68071f40 2024-03-29T11:17:53.219 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.238+0000 7fab65ffb640 1 -- 172.21.15.47:0/569620098 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1270812516 0 0) 0x7fab680aec40 con 0x7fab680adcb0 2024-03-29T11:17:53.219 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.238+0000 7fab65ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:17:53.219 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.238+0000 7fab65ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:53.219 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.238+0000 7fab65ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:53.219 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.238+0000 7fab65ffb640 10 monclient(hunting): my global_id is 4436 2024-03-29T11:17:53.219 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.238+0000 7fab65ffb640 1 -- 172.21.15.47:0/569620098 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab680af000 con 0x7fab680adcb0 2024-03-29T11:17:53.219 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.238+0000 7fab65ffb640 1 -- 172.21.15.47:0/569620098 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3066524230 0 0) 0x7fab54003520 con 0x7fab68071f40 2024-03-29T11:17:53.219 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.238+0000 7fab65ffb640 1 -- 172.21.15.47:0/569620098 >> v1:172.21.15.47:6790/0 conn(0x7fab680adcb0 legacy=0x7fab68145560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:53.219 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.238+0000 7fab65ffb640 1 -- 172.21.15.47:0/569620098 >> v1:172.21.15.121:6789/0 conn(0x7fab6813f910 legacy=0x7fab68145c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:17:53.219 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.238+0000 7fab65ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:17:53.219 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.238+0000 7fab65ffb640 1 monclient: found mon.noname-a 2024-03-29T11:17:53.219 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.238+0000 7fab65ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:53.219 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.238+0000 7fab65ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-29T11:17:53.220 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.238+0000 7fab65ffb640 1 -- 172.21.15.47:0/569620098 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab680b0030 con 0x7fab68071f40 2024-03-29T11:17:53.220 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.238+0000 7fab65ffb640 10 monclient: _finish_auth 0 2024-03-29T11:17:53.220 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.238+0000 7fab65ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:17:53.220 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.238+0000 7fab65ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:53.220 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.238+0000 7fab65ffb640 1 -- 172.21.15.47:0/569620098 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fab50001690 con 0x7fab68071f40 2024-03-29T11:17:53.220 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.238+0000 7fab65ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:53.220 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.238+0000 7fab65ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:17:53.220 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.238+0000 7fab65ffb640 10 monclient: dump: 2024-03-29T11:17:53.220 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:17:53.220 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:17:53.220 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:53.220 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:53.220 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:53.220 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:17:53.220 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:17:53.220 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:17:53.221 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:17:53.221 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:17:53.221 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.239+0000 7fab65ffb640 1 -- 172.21.15.47:0/569620098 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fab500018b0 con 0x7fab68071f40 2024-03-29T11:17:53.221 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.239+0000 7fab65ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:53.221 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.239+0000 7fab65ffb640 1 -- 172.21.15.47:0/569620098 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fab50001c40 con 0x7fab68071f40 2024-03-29T11:17:53.221 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.239+0000 7fab65ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:53.221 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.239+0000 7fab65ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:17:53.221 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.239+0000 7fab65ffb640 10 monclient: dump: 2024-03-29T11:17:53.221 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:17:53.221 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:17:53.221 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:53.221 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:53.221 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:53.221 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:17:53.222 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:17:53.222 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:17:53.222 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:17:53.222 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:17:53.222 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.239+0000 7fab67fff640 5 monclient: authenticate success, global_id 4380 2024-03-29T11:17:53.222 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.239+0000 7fab67fff640 10 monclient: get_monmap_and_config success 2024-03-29T11:17:53.222 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.239+0000 7fab67fff640 4 set_mon_vals no callback set 2024-03-29T11:17:53.222 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.240+0000 7fab67fff640 10 monclient: shutdown 2024-03-29T11:17:53.222 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.240+0000 7fab67fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:53.222 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.240+0000 7fab67fff640 1 -- 172.21.15.47:0/569620098 >> v1:172.21.15.47:6789/0 conn(0x7fab68071f40 legacy=0x7fab6813f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:53.222 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.240+0000 7fab67fff640 1 -- 172.21.15.47:0/569620098 shutdown_connections 2024-03-29T11:17:53.222 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.240+0000 7fab67fff640 1 -- 172.21.15.47:0/569620098 >> 172.21.15.47:0/569620098 conn(0x7fab680673e0 msgr2=0x7fab680b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:53.222 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.241+0000 7fab67fff640 1 -- 172.21.15.47:0/569620098 shutdown_connections 2024-03-29T11:17:53.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.244+0000 7fab67fff640 1 -- 172.21.15.47:0/569620098 wait complete. 2024-03-29T11:17:53.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.245+0000 7fab67fff640 10 monclient: build_initial_monmap 2024-03-29T11:17:53.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.245+0000 7fab67fff640 10 monclient: monmap: 2024-03-29T11:17:53.224 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:17:53.224 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:17:53.224 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:17:53.224 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:17:53.224 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:53.224 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:17:53.224 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:17:53.224 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:17:53.224 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:17:53.224 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:17:53.225 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.246+0000 7fab67fff640 1 librados: starting msgr at 2024-03-29T11:17:53.225 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.246+0000 7fab67fff640 1 librados: starting objecter 2024-03-29T11:17:53.225 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.246+0000 7fab67fff640 1 Processor -- start 2024-03-29T11:17:53.225 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.246+0000 7fab67fff640 1 -- start start 2024-03-29T11:17:53.225 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.247+0000 7fab67fff640 1 librados: setting wanted keys 2024-03-29T11:17:53.225 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.247+0000 7fab67fff640 1 librados: calling monclient init 2024-03-29T11:17:53.225 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.247+0000 7fab67fff640 10 monclient: init 2024-03-29T11:17:53.225 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.247+0000 7fab67fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:53.225 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.247+0000 7fab67fff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:17:53.225 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.247+0000 7fab67fff640 10 monclient(hunting): picked mon.noname-b con 0x7fab68071f40 addr v1:172.21.15.121:6789/0 2024-03-29T11:17:53.226 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.247+0000 7fab67fff640 10 monclient(hunting): picked mon.noname-c con 0x7fab680adcb0 addr v1:172.21.15.47:6790/0 2024-03-29T11:17:53.226 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.247+0000 7fab67fff640 10 monclient(hunting): picked mon.noname-a con 0x7fab6813f910 addr v1:172.21.15.47:6789/0 2024-03-29T11:17:53.226 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.247+0000 7fab67fff640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab68146330 con 0x7fab6813f910 2024-03-29T11:17:53.226 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.247+0000 7fab67fff640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab68130b20 con 0x7fab68071f40 2024-03-29T11:17:53.226 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.247+0000 7fab67fff640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab68145420 con 0x7fab680adcb0 2024-03-29T11:17:53.226 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.247+0000 7fab67fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:53.226 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.247+0000 7fab67fff640 10 monclient(hunting): authenticate will time out at 1116.435746s 2024-03-29T11:17:53.227 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.250+0000 7fab667fc640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fab680adcb0 0x7fab68130f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:54842/0 (socket says 172.21.15.47:54842) 2024-03-29T11:17:53.227 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.250+0000 7fab667fc640 1 -- 172.21.15.47:0/2505782186 learned_addr learned my addr 172.21.15.47:0/2505782186 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:17:53.231 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.254+0000 7fab5f7fe640 10 client.?.objecter ms_handle_connect 0x7fab680adcb0 2024-03-29T11:17:53.231 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.254+0000 7fab5f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:53.231 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.254+0000 7fab5f7fe640 10 client.?.objecter ms_handle_connect 0x7fab6813f910 2024-03-29T11:17:53.231 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.254+0000 7fab5f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:53.231 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.254+0000 7fab5f7fe640 1 -- 172.21.15.47:0/2505782186 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 857353984 0 0) 0x7fab68145420 con 0x7fab680adcb0 2024-03-29T11:17:53.231 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.254+0000 7fab5f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:17:53.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.254+0000 7fab5f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:53.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.254+0000 7fab5f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:53.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.254+0000 7fab5f7fe640 10 monclient(hunting): my global_id is 4439 2024-03-29T11:17:53.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.254+0000 7fab5f7fe640 1 -- 172.21.15.47:0/2505782186 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab48004eb0 con 0x7fab680adcb0 2024-03-29T11:17:53.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.254+0000 7fab5f7fe640 1 -- 172.21.15.47:0/2505782186 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2895877577 0 0) 0x7fab68146330 con 0x7fab6813f910 2024-03-29T11:17:53.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.254+0000 7fab5f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:17:53.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.254+0000 7fab5f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:53.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.254+0000 7fab5f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:53.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.254+0000 7fab5f7fe640 10 monclient(hunting): my global_id is 4383 2024-03-29T11:17:53.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.254+0000 7fab5f7fe640 1 -- 172.21.15.47:0/2505782186 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab68145420 con 0x7fab6813f910 2024-03-29T11:17:53.234 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.255+0000 7fab5f7fe640 10 client.?.objecter ms_handle_connect 0x7fab68071f40 2024-03-29T11:17:53.234 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.255+0000 7fab5f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:53.234 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.255+0000 7fab5f7fe640 1 -- 172.21.15.47:0/2505782186 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2093066672 0 0) 0x7fab68145420 con 0x7fab6813f910 2024-03-29T11:17:53.234 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.255+0000 7fab5f7fe640 1 -- 172.21.15.47:0/2505782186 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fab68146330 con 0x7fab6813f910 2024-03-29T11:17:53.234 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.255+0000 7fab5f7fe640 1 -- 172.21.15.47:0/2505782186 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fab58001640 con 0x7fab6813f910 2024-03-29T11:17:53.234 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.255+0000 7fab5f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:53.234 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.255+0000 7fab5f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:17:53.234 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.255+0000 7fab5f7fe640 10 monclient(hunting): dump: 2024-03-29T11:17:53.234 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:17:53.234 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:17:53.235 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:53.235 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:53.235 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:53.235 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:17:53.235 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:17:53.235 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:17:53.235 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:17:53.235 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:17:53.235 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.255+0000 7fab5f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:17:53.235 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.256+0000 7fab5f7fe640 1 -- 172.21.15.47:0/2505782186 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3678447414 0 0) 0x7fab68146330 con 0x7fab6813f910 2024-03-29T11:17:53.235 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.256+0000 7fab5f7fe640 1 -- 172.21.15.47:0/2505782186 >> v1:172.21.15.47:6790/0 conn(0x7fab680adcb0 legacy=0x7fab68130f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:53.235 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.256+0000 7fab5f7fe640 1 -- 172.21.15.47:0/2505782186 >> v1:172.21.15.121:6789/0 conn(0x7fab68071f40 legacy=0x7fab68144b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:53.235 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.256+0000 7fab5f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:17:53.236 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.256+0000 7fab5f7fe640 1 monclient: found mon.a 2024-03-29T11:17:53.236 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.256+0000 7fab5f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:53.236 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.256+0000 7fab5f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:17:53.236 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.256+0000 7fab5f7fe640 1 -- 172.21.15.47:0/2505782186 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab68131650 con 0x7fab6813f910 2024-03-29T11:17:53.236 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.256+0000 7fab5f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:17:53.236 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.256+0000 7fab5f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:17:53.236 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.256+0000 7fab5f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:53.236 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.256+0000 7fab5f7fe640 1 -- 172.21.15.47:0/2505782186 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fab58001b50 con 0x7fab6813f910 2024-03-29T11:17:53.236 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.256+0000 7fab5f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:53.236 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.256+0000 7fab5f7fe640 1 -- 172.21.15.47:0/2505782186 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fab580035f0 con 0x7fab6813f910 2024-03-29T11:17:53.236 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.256+0000 7fab5f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:53.236 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.256+0000 7fab5f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:17:53.236 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.256+0000 7fab5f7fe640 10 monclient: dump: 2024-03-29T11:17:53.236 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:17:53.237 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:17:53.237 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:53.237 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:17:53.237 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:53.237 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:17:53.237 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:17:53.237 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:17:53.237 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:17:53.237 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:17:53.239 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.258+0000 7fab67fff640 5 monclient: authenticate success, global_id 4383 2024-03-29T11:17:53.240 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.258+0000 7fab67fff640 10 monclient: _renew_subs 2024-03-29T11:17:53.240 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.258+0000 7fab67fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:17:53.240 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.258+0000 7fab67fff640 1 -- 172.21.15.47:0/2505782186 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab68143fb0 con 0x7fab6813f910 2024-03-29T11:17:53.240 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.258+0000 7fab67fff640 10 client.4383.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:17:53.240 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.258+0000 7fab67fff640 10 monclient: _renew_subs 2024-03-29T11:17:53.240 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.258+0000 7fab67fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:17:53.240 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.258+0000 7fab67fff640 1 -- 172.21.15.47:0/2505782186 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fab6813e620 con 0x7fab6813f910 2024-03-29T11:17:53.240 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.258+0000 7fab67fff640 1 librados: init done 2024-03-29T11:17:53.240 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.259+0000 7fab64ff9640 4 set_mon_vals no callback set 2024-03-29T11:17:53.240 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.260+0000 7fab5d7fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:53.240 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.260+0000 7fab5d7fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:53.240 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.260+0000 7fab5d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:17:53.240 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.260+0000 7fab5d7fa640 1 -- 172.21.15.47:0/2505782186 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab68145290 con 0x7fab6813f910 2024-03-29T11:17:53.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.261+0000 7fab5f7fe640 1 -- 172.21.15.47:0/2505782186 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7fab580195c0 con 0x7fab6813f910 2024-03-29T11:17:53.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.262+0000 7fab66ffd640 10 monclient: get_auth_request con 0x7fab4806ddc0 auth_method 0 2024-03-29T11:17:53.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.262+0000 7fab5f7fe640 1 -- 172.21.15.47:0/2505782186 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7fab580835b0 con 0x7fab6813f910 2024-03-29T11:17:53.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.262+0000 7fab5f7fe640 10 client.4383.objecter ms_dispatch 0x7fab68000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:17:53.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.262+0000 7fab5f7fe640 3 client.4383.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:17:53.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.262+0000 7fab5f7fe640 3 client.4383.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:17:53.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.262+0000 7fab5f7fe640 20 client.4383.objecter dump_active .. 0 homeless 2024-03-29T11:17:53.241 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.262+0000 7fab5f7fe640 10 client.4383.objecter ms_handle_connect 0x7fab4806ddc0 2024-03-29T11:17:53.246 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.269+0000 7fab5f7fe640 1 -- 172.21.15.47:0/2505782186 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fab5804ecc0 con 0x7fab6813f910 2024-03-29T11:17:53.246 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.269+0000 7fab5f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:53.246 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.269+0000 7fab5f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:17:53.585 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.606+0000 7fab5d7fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:53.585 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.606+0000 7fab5d7fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:53.585 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.606+0000 7fab5d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:17:53.585 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.606+0000 7fab5d7fa640 1 -- 172.21.15.47:0/2505782186 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fab6804f730 con 0x7fab6813f910 2024-03-29T11:17:53.585 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.607+0000 7fab5f7fe640 1 -- 172.21.15.47:0/2505782186 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7fab58050dc0 con 0x7fab6813f910 2024-03-29T11:17:53.585 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.607+0000 7fab5f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:53.585 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.607+0000 7fab5f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:17:53.586 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:17:53.586 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:17:53.588 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.610+0000 7fab5d7fa640 10 librados: watch_flush enter 2024-03-29T11:17:53.588 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.611+0000 7fab5d7fa640 10 librados: watch_flush exit 2024-03-29T11:17:53.588 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.611+0000 7fab5d7fa640 20 client.4383.objecter shutdown clearing up homeless session... 2024-03-29T11:17:53.588 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.611+0000 7fab5d7fa640 10 client.4383.objecter successfully canceled tick 2024-03-29T11:17:53.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.611+0000 7fab5d7fa640 1 -- 172.21.15.47:0/2505782186 >> v1:172.21.15.121:6816/1964117370 conn(0x7fab4806ddc0 legacy=0x7fab48070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:53.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.611+0000 7fab5d7fa640 10 monclient: shutdown 2024-03-29T11:17:53.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.611+0000 7fab5d7fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:53.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.611+0000 7fab5d7fa640 1 -- 172.21.15.47:0/2505782186 >> v1:172.21.15.47:6789/0 conn(0x7fab6813f910 legacy=0x7fab681e1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:53.590 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.612+0000 7fab5d7fa640 1 -- 172.21.15.47:0/2505782186 shutdown_connections 2024-03-29T11:17:53.590 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.612+0000 7fab5d7fa640 1 -- 172.21.15.47:0/2505782186 >> 172.21.15.47:0/2505782186 conn(0x7fab680673e0 msgr2=0x7fab68069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:53.590 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.612+0000 7fab5d7fa640 1 -- 172.21.15.47:0/2505782186 shutdown_connections 2024-03-29T11:17:53.590 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.612+0000 7fab5d7fa640 1 -- 172.21.15.47:0/2505782186 wait complete. 2024-03-29T11:17:53.590 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:17:53.612+0000 7fab5d7fa640 1 librados: shutdown 2024-03-29T11:17:53.606 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:18:00.414 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:18:00.558 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.581+0000 7f299aa84640 10 monclient: get_monmap_and_config 2024-03-29T11:18:00.558 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.581+0000 7f299aa84640 10 monclient: build_initial_monmap 2024-03-29T11:18:00.558 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.581+0000 7f299aa84640 10 monclient: monmap: 2024-03-29T11:18:00.558 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:18:00.558 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:18:00.559 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:18:00.582381+0000 2024-03-29T11:18:00.559 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:18:00.582381+0000 2024-03-29T11:18:00.559 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:00.559 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:00.559 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:18:00.559 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:18:00.559 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:18:00.559 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:00.560 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.582+0000 7f299aa84640 1 Processor -- start 2024-03-29T11:18:00.560 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.582+0000 7f299aa84640 1 -- start start 2024-03-29T11:18:00.560 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.582+0000 7f299aa84640 10 monclient: init 2024-03-29T11:18:00.562 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.583+0000 7f299aa84640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:00.562 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.583+0000 7f299aa84640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:18:00.562 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.583+0000 7f299aa84640 10 monclient(hunting): picked mon.noname-c con 0x7f29940b4820 addr v1:172.21.15.121:6789/0 2024-03-29T11:18:00.562 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.585+0000 7f299aa84640 10 monclient(hunting): picked mon.noname-b con 0x7f29940a9ce0 addr v1:172.21.15.47:6790/0 2024-03-29T11:18:00.562 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.585+0000 7f299aa84640 10 monclient(hunting): picked mon.noname-a con 0x7f29940b3cd0 addr v1:172.21.15.47:6789/0 2024-03-29T11:18:00.562 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.585+0000 7f299aa84640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29940b4180 con 0x7f29940b3cd0 2024-03-29T11:18:00.562 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.585+0000 7f299aa84640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29940aaf60 con 0x7f29940b4820 2024-03-29T11:18:00.562 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.585+0000 7f299aa84640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f299413c200 con 0x7f29940a9ce0 2024-03-29T11:18:00.563 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.585+0000 7f299aa84640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:00.563 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.585+0000 7f299aa84640 10 monclient(hunting): authenticate will time out at 1123.773419s 2024-03-29T11:18:00.563 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.586+0000 7f299a283640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f29940b3cd0 0x7f299414d7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:43832/0 (socket says 172.21.15.47:43832) 2024-03-29T11:18:00.563 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.586+0000 7f299a283640 1 -- 172.21.15.47:0/3649555009 learned_addr learned my addr 172.21.15.47:0/3649555009 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:18:00.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.588+0000 7f2998a80640 1 -- 172.21.15.47:0/3649555009 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3331203640 0 0) 0x7f2990001110 con 0x7f29940b3cd0 2024-03-29T11:18:00.566 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.588+0000 7f2998a80640 10 monclient(hunting): _negotiate 2024-03-29T11:18:00.566 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.588+0000 7f2998a80640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:00.566 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.588+0000 7f2998a80640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:00.566 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.588+0000 7f2998a80640 10 monclient(hunting): my global_id is 4386 2024-03-29T11:18:00.566 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.588+0000 7f2998a80640 1 -- 172.21.15.47:0/3649555009 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2980003520 con 0x7f29940b3cd0 2024-03-29T11:18:00.566 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.589+0000 7f2998a80640 1 -- 172.21.15.47:0/3649555009 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4190708938 0 0) 0x7f2980003520 con 0x7f29940b3cd0 2024-03-29T11:18:00.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.589+0000 7f2998a80640 1 -- 172.21.15.47:0/3649555009 >> v1:172.21.15.47:6790/0 conn(0x7f29940a9ce0 legacy=0x7f2994145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:18:00.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.589+0000 7f2998a80640 1 -- 172.21.15.47:0/3649555009 >> v1:172.21.15.121:6789/0 conn(0x7f29940b4820 legacy=0x7f299413baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:00.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.589+0000 7f2998a80640 10 monclient: _finish_hunting 0 2024-03-29T11:18:00.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.589+0000 7f2998a80640 1 monclient: found mon.noname-a 2024-03-29T11:18:00.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.589+0000 7f2998a80640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:00.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.589+0000 7f2998a80640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-29T11:18:00.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.589+0000 7f2998a80640 1 -- 172.21.15.47:0/3649555009 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2994145e80 con 0x7f29940b3cd0 2024-03-29T11:18:00.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.589+0000 7f2998a80640 10 monclient: _finish_auth 0 2024-03-29T11:18:00.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.589+0000 7f2998a80640 10 monclient: _check_auth_tickets 2024-03-29T11:18:00.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.589+0000 7f2998a80640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:00.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.589+0000 7f2998a80640 1 -- 172.21.15.47:0/3649555009 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f2990001660 con 0x7f29940b3cd0 2024-03-29T11:18:00.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.589+0000 7f2998a80640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:00.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.589+0000 7f2998a80640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:18:00.568 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.589+0000 7f2998a80640 10 monclient: dump: 2024-03-29T11:18:00.568 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:18:00.568 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:18:00.568 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:00.568 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:00.568 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:00.568 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:00.568 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:18:00.568 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:18:00.568 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:18:00.568 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:00.568 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.589+0000 7f2998a80640 1 -- 172.21.15.47:0/3649555009 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f29900018f0 con 0x7f29940b3cd0 2024-03-29T11:18:00.568 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.589+0000 7f2998a80640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:00.568 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.589+0000 7f2998a80640 1 -- 172.21.15.47:0/3649555009 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f2990001660 con 0x7f29940b3cd0 2024-03-29T11:18:00.569 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.589+0000 7f2998a80640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:00.569 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.589+0000 7f2998a80640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:18:00.569 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.589+0000 7f2998a80640 10 monclient: dump: 2024-03-29T11:18:00.569 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:18:00.569 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:18:00.569 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:00.569 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:00.569 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:00.569 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:00.569 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:18:00.569 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:18:00.569 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:18:00.569 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:00.570 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.589+0000 7f299aa84640 5 monclient: authenticate success, global_id 4386 2024-03-29T11:18:00.570 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.589+0000 7f299aa84640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:00.570 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.589+0000 7f299aa84640 4 set_mon_vals no callback set 2024-03-29T11:18:00.571 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.590+0000 7f299aa84640 10 monclient: shutdown 2024-03-29T11:18:00.571 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.590+0000 7f299aa84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:00.571 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.590+0000 7f299aa84640 1 -- 172.21.15.47:0/3649555009 >> v1:172.21.15.47:6789/0 conn(0x7f29940b3cd0 legacy=0x7f299414d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:00.571 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.590+0000 7f299aa84640 1 -- 172.21.15.47:0/3649555009 shutdown_connections 2024-03-29T11:18:00.571 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.590+0000 7f299aa84640 1 -- 172.21.15.47:0/3649555009 >> 172.21.15.47:0/3649555009 conn(0x7f299406d070 msgr2=0x7f2994066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:00.571 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.591+0000 7f299aa84640 1 -- 172.21.15.47:0/3649555009 shutdown_connections 2024-03-29T11:18:00.572 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.591+0000 7f299aa84640 1 -- 172.21.15.47:0/3649555009 wait complete. 2024-03-29T11:18:00.572 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.592+0000 7f299aa84640 10 monclient: build_initial_monmap 2024-03-29T11:18:00.572 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.592+0000 7f299aa84640 10 monclient: monmap: 2024-03-29T11:18:00.572 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:18:00.572 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:00.572 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:18:00.572 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:18:00.572 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:00.572 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:00.572 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:18:00.572 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:18:00.572 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:18:00.572 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:00.572 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.592+0000 7f299aa84640 1 librados: starting msgr at 2024-03-29T11:18:00.572 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.592+0000 7f299aa84640 1 librados: starting objecter 2024-03-29T11:18:00.573 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.593+0000 7f299aa84640 1 Processor -- start 2024-03-29T11:18:00.573 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.593+0000 7f299aa84640 1 -- start start 2024-03-29T11:18:00.573 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.593+0000 7f299aa84640 1 librados: setting wanted keys 2024-03-29T11:18:00.573 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.593+0000 7f299aa84640 1 librados: calling monclient init 2024-03-29T11:18:00.573 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.593+0000 7f299aa84640 10 monclient: init 2024-03-29T11:18:00.573 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.594+0000 7f299aa84640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:00.573 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.594+0000 7f299aa84640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:18:00.573 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.594+0000 7f299aa84640 10 monclient(hunting): picked mon.noname-c con 0x7f29940a9ce0 addr v1:172.21.15.47:6790/0 2024-03-29T11:18:00.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.597+0000 7f2999a82640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f29940a9ce0 0x7f2994130da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:33939/0 (socket says 172.21.15.47:33939) 2024-03-29T11:18:00.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.597+0000 7f299aa84640 10 monclient(hunting): picked mon.noname-b con 0x7f29940b3cd0 addr v1:172.21.15.121:6789/0 2024-03-29T11:18:00.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.597+0000 7f299aa84640 10 monclient(hunting): picked mon.noname-a con 0x7f29940b4820 addr v1:172.21.15.47:6789/0 2024-03-29T11:18:00.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.597+0000 7f299aa84640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f299406e450 con 0x7f29940b4820 2024-03-29T11:18:00.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.597+0000 7f299aa84640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f299406f1a0 con 0x7f29940b3cd0 2024-03-29T11:18:00.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.597+0000 7f299aa84640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f299406f480 con 0x7f29940a9ce0 2024-03-29T11:18:00.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.597+0000 7f299aa84640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:00.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.597+0000 7f299aa84640 10 monclient(hunting): authenticate will time out at 1123.785200s 2024-03-29T11:18:00.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.597+0000 7f2999a82640 1 -- 172.21.15.47:0/1906727481 learned_addr learned my addr 172.21.15.47:0/1906727481 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:18:00.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.597+0000 7f298a7fc640 10 client.?.objecter ms_handle_connect 0x7f29940a9ce0 2024-03-29T11:18:00.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.597+0000 7f298a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:00.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.597+0000 7f298a7fc640 10 client.?.objecter ms_handle_connect 0x7f29940b4820 2024-03-29T11:18:00.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.598+0000 7f298a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:00.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.598+0000 7f298a7fc640 1 -- 172.21.15.47:0/1906727481 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2558455926 0 0) 0x7f299406f480 con 0x7f29940a9ce0 2024-03-29T11:18:00.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.598+0000 7f298a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:18:00.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.598+0000 7f298a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:00.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.598+0000 7f298a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:00.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.598+0000 7f298a7fc640 10 monclient(hunting): my global_id is 4442 2024-03-29T11:18:00.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.598+0000 7f298a7fc640 1 -- 172.21.15.47:0/1906727481 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2978004eb0 con 0x7f29940a9ce0 2024-03-29T11:18:00.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.598+0000 7f298a7fc640 10 client.?.objecter ms_handle_connect 0x7f29940b3cd0 2024-03-29T11:18:00.580 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.598+0000 7f298a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:00.580 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.598+0000 7f298a7fc640 1 -- 172.21.15.47:0/1906727481 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1326069033 0 0) 0x7f299406f1a0 con 0x7f29940b3cd0 2024-03-29T11:18:00.580 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.598+0000 7f298a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:18:00.580 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.598+0000 7f298a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:00.580 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.598+0000 7f298a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:00.580 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.598+0000 7f298a7fc640 10 monclient(hunting): my global_id is 4399 2024-03-29T11:18:00.580 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.598+0000 7f298a7fc640 1 -- 172.21.15.47:0/1906727481 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f299406f480 con 0x7f29940b3cd0 2024-03-29T11:18:00.580 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.598+0000 7f298a7fc640 1 -- 172.21.15.47:0/1906727481 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3557480455 0 0) 0x7f2978004eb0 con 0x7f29940a9ce0 2024-03-29T11:18:00.580 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.598+0000 7f298a7fc640 1 -- 172.21.15.47:0/1906727481 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f299406f1a0 con 0x7f29940a9ce0 2024-03-29T11:18:00.580 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.598+0000 7f298a7fc640 1 -- 172.21.15.47:0/1906727481 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f2984001950 con 0x7f29940a9ce0 2024-03-29T11:18:00.580 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.598+0000 7f298a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:00.580 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.598+0000 7f298a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:18:00.580 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.598+0000 7f298a7fc640 10 monclient(hunting): dump: 2024-03-29T11:18:00.580 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:18:00.580 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:18:00.580 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:00.581 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:00.581 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:00.581 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:00.581 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:18:00.581 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:18:00.581 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:18:00.581 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:00.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.598+0000 7f298a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:00.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.598+0000 7f298a7fc640 1 -- 172.21.15.47:0/1906727481 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3848499606 0 0) 0x7f299406e450 con 0x7f29940b4820 2024-03-29T11:18:00.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.598+0000 7f298a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:18:00.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.598+0000 7f298a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:00.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.598+0000 7f298a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:00.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.598+0000 7f298a7fc640 10 monclient(hunting): my global_id is 4389 2024-03-29T11:18:00.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.598+0000 7f298a7fc640 1 -- 172.21.15.47:0/1906727481 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2978004eb0 con 0x7f29940b4820 2024-03-29T11:18:00.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.598+0000 7f298a7fc640 1 -- 172.21.15.47:0/1906727481 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3835869092 0 0) 0x7f299406f480 con 0x7f29940b3cd0 2024-03-29T11:18:00.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.598+0000 7f298a7fc640 1 -- 172.21.15.47:0/1906727481 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f299406e450 con 0x7f29940b3cd0 2024-03-29T11:18:00.582 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.598+0000 7f298a7fc640 1 -- 172.21.15.47:0/1906727481 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f298c001890 con 0x7f29940b3cd0 2024-03-29T11:18:00.582 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.598+0000 7f298a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:00.582 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.598+0000 7f298a7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:18:00.582 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.598+0000 7f298a7fc640 10 monclient(hunting): dump: 2024-03-29T11:18:00.582 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:18:00.582 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:18:00.582 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:00.582 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:00.582 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:00.582 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:00.582 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:18:00.582 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:18:00.582 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:18:00.582 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:00.582 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.598+0000 7f298a7fc640 1 -- 172.21.15.47:0/1906727481 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1015127280 0 0) 0x7f299406f1a0 con 0x7f29940a9ce0 2024-03-29T11:18:00.583 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.598+0000 7f298a7fc640 1 -- 172.21.15.47:0/1906727481 >> v1:172.21.15.121:6789/0 conn(0x7f29940b3cd0 legacy=0x7f29941448f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:00.583 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.598+0000 7f298a7fc640 1 -- 172.21.15.47:0/1906727481 >> v1:172.21.15.47:6789/0 conn(0x7f29940b4820 legacy=0x7f29941e6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:00.583 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.598+0000 7f298a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:18:00.583 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.598+0000 7f298a7fc640 1 monclient: found mon.c 2024-03-29T11:18:00.583 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.598+0000 7f298a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:00.583 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.598+0000 7f298a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:18:00.583 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.598+0000 7f298a7fc640 1 -- 172.21.15.47:0/1906727481 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f299406df30 con 0x7f29940a9ce0 2024-03-29T11:18:00.583 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.598+0000 7f298a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:18:00.583 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.598+0000 7f298a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:18:00.583 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.598+0000 7f298a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:00.583 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.598+0000 7f299aa84640 5 monclient: authenticate success, global_id 4442 2024-03-29T11:18:00.583 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.598+0000 7f299aa84640 10 monclient: _renew_subs 2024-03-29T11:18:00.583 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.598+0000 7f299aa84640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:18:00.583 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.598+0000 7f299aa84640 1 -- 172.21.15.47:0/1906727481 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f299412cb90 con 0x7f29940a9ce0 2024-03-29T11:18:00.583 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.598+0000 7f299aa84640 10 client.4442.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:00.583 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.598+0000 7f299aa84640 10 monclient: _renew_subs 2024-03-29T11:18:00.584 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.598+0000 7f299aa84640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:18:00.584 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.598+0000 7f299aa84640 1 -- 172.21.15.47:0/1906727481 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2994145000 con 0x7f29940a9ce0 2024-03-29T11:18:00.584 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.598+0000 7f299aa84640 1 librados: init done 2024-03-29T11:18:00.584 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.600+0000 7f299aa84640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:00.584 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.600+0000 7f299aa84640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:00.584 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.600+0000 7f2999a82640 1 -- 172.21.15.47:0/1906727481 >> v1:172.21.15.47:6790/0 conn(0x7f29940a9ce0 legacy=0x7f2994130da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-29T11:18:00.584 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.600+0000 7f2999a82640 1 -- 172.21.15.47:0/1906727481 >> v1:172.21.15.47:6790/0 conn(0x7f29940a9ce0 legacy=0x7f2994130da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:18:00.584 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.600+0000 7f2999a82640 1 --1- 172.21.15.47:0/1906727481 >> v1:172.21.15.47:6790/0 conn(0x7f29940a9ce0 0x7f2994130da0 :-1 s=OPENED pgs=522716 cs=1 l=1).handle_message read tag failed 2024-03-29T11:18:00.584 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.600+0000 7f2999a82640 1 --1- 172.21.15.47:0/1906727481 >> v1:172.21.15.47:6790/0 conn(0x7f29940a9ce0 0x7f2994130da0 :-1 s=OPENED pgs=522716 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:18:00.584 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.600+0000 7f299aa84640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:18:00.584 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.600+0000 7f299aa84640 1 -- 172.21.15.47:0/1906727481 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f299404f730 con 0x7f29940a9ce0 2024-03-29T11:18:00.584 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.600+0000 7f298a7fc640 10 monclient: ms_handle_reset current mon v1:172.21.15.47:6790/0 2024-03-29T11:18:00.584 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.600+0000 7f298a7fc640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:00.584 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.600+0000 7f298a7fc640 1 -- 172.21.15.47:0/1906727481 >> v1:172.21.15.47:6790/0 conn(0x7f29940a9ce0 legacy=0x7f2994130da0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:18:00.584 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.600+0000 7f298a7fc640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:18:00.584 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.600+0000 7f298a7fc640 10 monclient(hunting): picked mon.c con 0x7f29780066b0 addr v1:172.21.15.47:6790/0 2024-03-29T11:18:00.585 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.600+0000 7f298a7fc640 10 monclient(hunting): picked mon.a con 0x7f29940b4820 addr v1:172.21.15.47:6789/0 2024-03-29T11:18:00.585 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.600+0000 7f298a7fc640 10 monclient(hunting): picked mon.b con 0x7f29940b3cd0 addr v1:172.21.15.121:6789/0 2024-03-29T11:18:00.585 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.600+0000 7f298a7fc640 1 -- 172.21.15.47:0/1906727481 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f299406f1a0 con 0x7f29940b4820 2024-03-29T11:18:00.585 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.600+0000 7f298a7fc640 1 -- 172.21.15.47:0/1906727481 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f299406f480 con 0x7f29940b3cd0 2024-03-29T11:18:00.585 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.600+0000 7f298a7fc640 1 -- 172.21.15.47:0/1906727481 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2978013d50 con 0x7f29780066b0 2024-03-29T11:18:00.585 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.600+0000 7f298a7fc640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:00.588 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.603+0000 7f298a7fc640 10 client.4442.objecter ms_handle_connect 0x7f29940b4820 2024-03-29T11:18:00.588 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.603+0000 7f298a7fc640 10 client.4442.objecter resend_mon_ops 2024-03-29T11:18:00.588 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.603+0000 7f298a7fc640 10 client.4442.objecter ms_handle_connect 0x7f29780066b0 2024-03-29T11:18:00.588 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.603+0000 7f298a7fc640 10 client.4442.objecter resend_mon_ops 2024-03-29T11:18:00.588 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.603+0000 7f298a7fc640 1 -- 172.21.15.47:0/1906727481 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2152164730 0 0) 0x7f299406f1a0 con 0x7f29940b4820 2024-03-29T11:18:00.588 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.603+0000 7f298a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:18:00.588 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.603+0000 7f298a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:00.588 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.603+0000 7f298a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-29T11:18:00.588 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.603+0000 7f298a7fc640 1 -- 172.21.15.47:0/1906727481 --> v1:172.21.15.47:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f2978013ee0 con 0x7f29940b4820 2024-03-29T11:18:00.588 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.606+0000 7f298a7fc640 1 -- 172.21.15.47:0/1906727481 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 2349187097 0 0) 0x7f2978013ee0 con 0x7f29940b4820 2024-03-29T11:18:00.588 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.606+0000 7f298a7fc640 1 -- 172.21.15.47:0/1906727481 >> v1:172.21.15.47:6790/0 conn(0x7f29780066b0 legacy=0x7f29780092f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:00.588 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.606+0000 7f298a7fc640 1 -- 172.21.15.47:0/1906727481 >> v1:172.21.15.121:6789/0 conn(0x7f29940b3cd0 legacy=0x7f2978010150 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:18:00.588 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.606+0000 7f298a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:18:00.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.606+0000 7f298a7fc640 1 monclient: found mon.a 2024-03-29T11:18:00.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.606+0000 7f298a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:00.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.606+0000 7f298a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:18:00.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.606+0000 7f298a7fc640 1 -- 172.21.15.47:0/1906727481 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7f2978015130 con 0x7f29940b4820 2024-03-29T11:18:00.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.606+0000 7f298a7fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:00.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.606+0000 7f298a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:18:00.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.606+0000 7f298a7fc640 1 -- 172.21.15.47:0/1906727481 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f299406f480 con 0x7f29940b4820 2024-03-29T11:18:00.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.606+0000 7f298a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:18:00.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.606+0000 7f298a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:18:00.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.606+0000 7f298a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:00.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.606+0000 7f298a7fc640 1 -- 172.21.15.47:0/1906727481 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f298c001df0 con 0x7f29940b4820 2024-03-29T11:18:00.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.606+0000 7f298a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:00.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.606+0000 7f298a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:18:00.589 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.606+0000 7f298a7fc640 10 monclient: dump: 2024-03-29T11:18:00.590 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:18:00.590 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:18:00.590 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:00.590 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:00.590 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:00.590 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:00.590 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:18:00.590 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:18:00.590 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:18:00.590 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:00.590 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.609+0000 7f298a7fc640 1 -- 172.21.15.47:0/1906727481 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f298c002010 con 0x7f29940b4820 2024-03-29T11:18:00.590 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.609+0000 7f298a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:00.590 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.609+0000 7f298bfff640 4 set_mon_vals no callback set 2024-03-29T11:18:00.591 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.609+0000 7f298a7fc640 1 -- 172.21.15.47:0/1906727481 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7f298c019b00 con 0x7f29940b4820 2024-03-29T11:18:00.591 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.610+0000 7f298a7fc640 1 -- 172.21.15.47:0/1906727481 <== mon.0 v1:172.21.15.47:6789/0 6 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7f298c083ab0 con 0x7f29940b4820 2024-03-29T11:18:00.591 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.610+0000 7f298a7fc640 10 client.4442.objecter ms_dispatch 0x7f2994000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:18:00.591 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.610+0000 7f298a7fc640 3 client.4442.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:18:00.591 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.610+0000 7f2999a82640 10 monclient: get_auth_request con 0x7f29940a9ce0 auth_method 0 2024-03-29T11:18:00.591 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.610+0000 7f298a7fc640 3 client.4442.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:18:00.591 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.611+0000 7f298a7fc640 20 client.4442.objecter dump_active .. 0 homeless 2024-03-29T11:18:00.591 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.611+0000 7f298a7fc640 10 client.4442.objecter ms_handle_connect 0x7f29940a9ce0 2024-03-29T11:18:00.596 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.619+0000 7f298a7fc640 1 -- 172.21.15.47:0/1906727481 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f298c04f450 con 0x7f29940b4820 2024-03-29T11:18:00.596 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.619+0000 7f298a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:00.596 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.619+0000 7f298a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:00.920 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.943+0000 7f299aa84640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:00.920 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.943+0000 7f299aa84640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:00.921 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.943+0000 7f299aa84640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:18:00.921 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.943+0000 7f299aa84640 1 -- 172.21.15.47:0/1906727481 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f299404f730 con 0x7f29940b4820 2024-03-29T11:18:00.923 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.946+0000 7f298a7fc640 1 -- 172.21.15.47:0/1906727481 <== mon.0 v1:172.21.15.47:6789/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7f298c051380 con 0x7f29940b4820 2024-03-29T11:18:00.923 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.946+0000 7f298a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:00.923 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.946+0000 7f298a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:00.923 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:18:00.924 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:18:00.927 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.950+0000 7f299aa84640 10 librados: watch_flush enter 2024-03-29T11:18:00.927 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.950+0000 7f299aa84640 10 librados: watch_flush exit 2024-03-29T11:18:00.927 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.950+0000 7f299aa84640 20 client.4442.objecter shutdown clearing up homeless session... 2024-03-29T11:18:00.927 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.950+0000 7f299aa84640 10 client.4442.objecter successfully canceled tick 2024-03-29T11:18:00.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.950+0000 7f299aa84640 1 -- 172.21.15.47:0/1906727481 >> v1:172.21.15.121:6816/1964117370 conn(0x7f29940a9ce0 legacy=0x7f297807bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:00.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.950+0000 7f299aa84640 10 monclient: shutdown 2024-03-29T11:18:00.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.950+0000 7f299aa84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:00.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.950+0000 7f299aa84640 1 -- 172.21.15.47:0/1906727481 >> v1:172.21.15.47:6789/0 conn(0x7f29940b4820 legacy=0x7f297800ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:00.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.950+0000 7f299aa84640 1 -- 172.21.15.47:0/1906727481 shutdown_connections 2024-03-29T11:18:00.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.950+0000 7f299aa84640 1 -- 172.21.15.47:0/1906727481 >> 172.21.15.47:0/1906727481 conn(0x7f299406d070 msgr2=0x7f2994141f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:00.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.950+0000 7f299aa84640 1 -- 172.21.15.47:0/1906727481 shutdown_connections 2024-03-29T11:18:00.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.950+0000 7f299aa84640 1 -- 172.21.15.47:0/1906727481 wait complete. 2024-03-29T11:18:00.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:00.951+0000 7f299aa84640 1 librados: shutdown 2024-03-29T11:18:00.945 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:18:07.752 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:18:07.933 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.956+0000 7f23cd89d640 10 monclient: get_monmap_and_config 2024-03-29T11:18:07.933 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.956+0000 7f23cd89d640 10 monclient: build_initial_monmap 2024-03-29T11:18:07.933 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.956+0000 7f23cd89d640 10 monclient: monmap: 2024-03-29T11:18:07.933 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:18:07.933 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:18:07.933 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:18:07.957010+0000 2024-03-29T11:18:07.933 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:18:07.957010+0000 2024-03-29T11:18:07.933 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:07.933 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:07.934 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:18:07.934 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:18:07.934 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:18:07.934 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:07.935 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.958+0000 7f23cd89d640 1 Processor -- start 2024-03-29T11:18:07.935 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.959+0000 7f23cd89d640 1 -- start start 2024-03-29T11:18:07.936 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.959+0000 7f23cd89d640 10 monclient: init 2024-03-29T11:18:07.937 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.960+0000 7f23cd89d640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:07.937 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.960+0000 7f23cd89d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:18:07.937 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.960+0000 7f23cd89d640 10 monclient(hunting): picked mon.noname-b con 0x7f23c81457f0 addr v1:172.21.15.47:6790/0 2024-03-29T11:18:07.937 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.960+0000 7f23cd89d640 10 monclient(hunting): picked mon.noname-c con 0x7f23c813f650 addr v1:172.21.15.121:6789/0 2024-03-29T11:18:07.938 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.960+0000 7f23cd89d640 10 monclient(hunting): picked mon.noname-a con 0x7f23c80a9d70 addr v1:172.21.15.47:6789/0 2024-03-29T11:18:07.938 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.960+0000 7f23cd89d640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23c80aaef0 con 0x7f23c80a9d70 2024-03-29T11:18:07.938 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.960+0000 7f23cd89d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23c8145bd0 con 0x7f23c813f650 2024-03-29T11:18:07.938 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.960+0000 7f23cd89d640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23c8145eb0 con 0x7f23c81457f0 2024-03-29T11:18:07.938 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.960+0000 7f23cd89d640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:07.938 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.960+0000 7f23cd89d640 10 monclient(hunting): authenticate will time out at 1131.148428s 2024-03-29T11:18:07.941 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.963+0000 7f23c67fc640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f23c813f650 0x7f23c80b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.47:59723/0 (socket says 172.21.15.47:59723) 2024-03-29T11:18:07.941 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.963+0000 7f23c67fc640 1 -- 172.21.15.47:0/111888967 learned_addr learned my addr 172.21.15.47:0/111888967 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:18:07.943 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.966+0000 7f23c5ffb640 1 -- 172.21.15.47:0/111888967 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2840932611 0 0) 0x7f23c8145bd0 con 0x7f23c813f650 2024-03-29T11:18:07.943 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.966+0000 7f23c5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:18:07.943 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.966+0000 7f23c5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:07.943 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.966+0000 7f23c5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:07.943 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.966+0000 7f23c5ffb640 10 monclient(hunting): my global_id is 4402 2024-03-29T11:18:07.944 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.966+0000 7f23c5ffb640 1 -- 172.21.15.47:0/111888967 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23b0003520 con 0x7f23c813f650 2024-03-29T11:18:07.947 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.970+0000 7f23c5ffb640 1 -- 172.21.15.47:0/111888967 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 540489092 0 0) 0x7f23c80aaef0 con 0x7f23c80a9d70 2024-03-29T11:18:07.947 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.970+0000 7f23c5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:18:07.948 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.970+0000 7f23c5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:07.948 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.970+0000 7f23c5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:07.948 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.970+0000 7f23c5ffb640 10 monclient(hunting): my global_id is 4392 2024-03-29T11:18:07.948 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.970+0000 7f23c5ffb640 1 -- 172.21.15.47:0/111888967 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23c8145bd0 con 0x7f23c80a9d70 2024-03-29T11:18:07.948 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.970+0000 7f23c5ffb640 1 -- 172.21.15.47:0/111888967 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4123680925 0 0) 0x7f23b0003520 con 0x7f23c813f650 2024-03-29T11:18:07.948 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.970+0000 7f23c5ffb640 1 -- 172.21.15.47:0/111888967 >> v1:172.21.15.47:6790/0 conn(0x7f23c81457f0 legacy=0x7f23c813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:07.948 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.970+0000 7f23c5ffb640 1 -- 172.21.15.47:0/111888967 >> v1:172.21.15.47:6789/0 conn(0x7f23c80a9d70 legacy=0x7f23c814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:07.948 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.970+0000 7f23c5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:18:07.948 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.970+0000 7f23c5ffb640 1 monclient: found mon.noname-c 2024-03-29T11:18:07.948 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.970+0000 7f23c5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:07.948 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.970+0000 7f23c5ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:18:07.948 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.970+0000 7f23c5ffb640 1 -- 172.21.15.47:0/111888967 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23c8146340 con 0x7f23c813f650 2024-03-29T11:18:07.948 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.970+0000 7f23c5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:18:07.948 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.970+0000 7f23c5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:18:07.948 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.970+0000 7f23c5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:07.949 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.970+0000 7f23c5ffb640 1 -- 172.21.15.47:0/111888967 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f23b8001610 con 0x7f23c813f650 2024-03-29T11:18:07.949 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.970+0000 7f23c5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:07.949 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.970+0000 7f23c5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:18:07.949 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.970+0000 7f23c5ffb640 10 monclient: dump: 2024-03-29T11:18:07.949 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:18:07.949 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:18:07.949 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:07.949 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:07.949 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:07.949 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:07.949 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:18:07.949 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:18:07.949 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:18:07.949 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:07.949 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.970+0000 7f23cd89d640 5 monclient: authenticate success, global_id 4402 2024-03-29T11:18:07.949 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.970+0000 7f23cd89d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:07.950 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.970+0000 7f23c5ffb640 1 -- 172.21.15.47:0/111888967 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23b8001770 con 0x7f23c813f650 2024-03-29T11:18:07.950 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.970+0000 7f23c5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:07.950 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.970+0000 7f23c5ffb640 1 -- 172.21.15.47:0/111888967 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f23b8001b80 con 0x7f23c813f650 2024-03-29T11:18:07.950 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.970+0000 7f23c5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:07.950 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.970+0000 7f23c5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:18:07.950 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.970+0000 7f23c5ffb640 10 monclient: dump: 2024-03-29T11:18:07.950 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:18:07.950 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:18:07.950 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:07.950 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:07.950 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:07.950 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:07.950 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:18:07.950 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:18:07.950 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:18:07.951 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:07.951 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.970+0000 7f23cd89d640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:07.951 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.970+0000 7f23cd89d640 4 set_mon_vals no callback set 2024-03-29T11:18:07.951 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.971+0000 7f23cd89d640 10 monclient: shutdown 2024-03-29T11:18:07.951 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.971+0000 7f23cd89d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:07.951 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.971+0000 7f23cd89d640 1 -- 172.21.15.47:0/111888967 >> v1:172.21.15.121:6789/0 conn(0x7f23c813f650 legacy=0x7f23c80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:07.951 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.971+0000 7f23cd89d640 1 -- 172.21.15.47:0/111888967 shutdown_connections 2024-03-29T11:18:07.952 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.971+0000 7f23cd89d640 1 -- 172.21.15.47:0/111888967 >> 172.21.15.47:0/111888967 conn(0x7f23c8066a00 msgr2=0x7f23c8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:07.952 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.973+0000 7f23cd89d640 1 -- 172.21.15.47:0/111888967 shutdown_connections 2024-03-29T11:18:07.952 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.975+0000 7f23cd89d640 1 -- 172.21.15.47:0/111888967 wait complete. 2024-03-29T11:18:07.954 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.976+0000 7f23cd89d640 10 monclient: build_initial_monmap 2024-03-29T11:18:07.954 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.976+0000 7f23cd89d640 10 monclient: monmap: 2024-03-29T11:18:07.954 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:18:07.954 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:07.954 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:18:07.955 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:18:07.955 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:07.955 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:07.955 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:18:07.955 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:18:07.955 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:18:07.955 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:07.955 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.976+0000 7f23cd89d640 1 librados: starting msgr at 2024-03-29T11:18:07.955 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.976+0000 7f23cd89d640 1 librados: starting objecter 2024-03-29T11:18:07.955 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.976+0000 7f23cd89d640 1 Processor -- start 2024-03-29T11:18:07.955 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.977+0000 7f23cd89d640 1 -- start start 2024-03-29T11:18:07.955 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.977+0000 7f23cd89d640 1 librados: setting wanted keys 2024-03-29T11:18:07.955 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.977+0000 7f23cd89d640 1 librados: calling monclient init 2024-03-29T11:18:07.956 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.977+0000 7f23cd89d640 10 monclient: init 2024-03-29T11:18:07.962 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.985+0000 7f23cd89d640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:07.962 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.985+0000 7f23cd89d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:18:07.967 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.987+0000 7f23cd89d640 10 monclient(hunting): picked mon.noname-c con 0x7f23c80a9d70 addr v1:172.21.15.47:6790/0 2024-03-29T11:18:07.967 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.987+0000 7f23cd89d640 10 monclient(hunting): picked mon.noname-b con 0x7f23c813f650 addr v1:172.21.15.121:6789/0 2024-03-29T11:18:07.967 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.987+0000 7f23cd89d640 10 monclient(hunting): picked mon.noname-a con 0x7f23c81457f0 addr v1:172.21.15.47:6789/0 2024-03-29T11:18:07.967 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.987+0000 7f23cd89d640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23c8137b50 con 0x7f23c81457f0 2024-03-29T11:18:07.968 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.987+0000 7f23cd89d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23c80ac4d0 con 0x7f23c813f650 2024-03-29T11:18:07.968 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.987+0000 7f23cd89d640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23c80ac7b0 con 0x7f23c80a9d70 2024-03-29T11:18:07.968 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.987+0000 7f23cd89d640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:07.968 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.987+0000 7f23cd89d640 10 monclient(hunting): authenticate will time out at 1131.175669s 2024-03-29T11:18:07.968 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.990+0000 7f23c6ffd640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f23c80a9d70 0x7f23c81425f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:53823/0 (socket says 172.21.15.47:53823) 2024-03-29T11:18:07.968 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.990+0000 7f23c6ffd640 1 -- 172.21.15.47:0/867693398 learned_addr learned my addr 172.21.15.47:0/867693398 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:18:07.969 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.990+0000 7f23affff640 10 client.?.objecter ms_handle_connect 0x7f23c80a9d70 2024-03-29T11:18:07.970 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.990+0000 7f23affff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:07.970 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.991+0000 7f23affff640 10 client.?.objecter ms_handle_connect 0x7f23c81457f0 2024-03-29T11:18:07.970 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.991+0000 7f23affff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:07.970 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.991+0000 7f23affff640 1 -- 172.21.15.47:0/867693398 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3934437097 0 0) 0x7f23c80ac7b0 con 0x7f23c80a9d70 2024-03-29T11:18:07.970 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.991+0000 7f23affff640 10 monclient(hunting): _negotiate 2024-03-29T11:18:07.970 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.991+0000 7f23affff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:07.970 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.991+0000 7f23affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:07.970 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.991+0000 7f23affff640 10 monclient(hunting): my global_id is 4445 2024-03-29T11:18:07.970 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.991+0000 7f23affff640 1 -- 172.21.15.47:0/867693398 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23b4004eb0 con 0x7f23c80a9d70 2024-03-29T11:18:07.970 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.991+0000 7f23affff640 1 -- 172.21.15.47:0/867693398 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3188281510 0 0) 0x7f23c8137b50 con 0x7f23c81457f0 2024-03-29T11:18:07.971 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.991+0000 7f23affff640 10 monclient(hunting): _negotiate 2024-03-29T11:18:07.971 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.991+0000 7f23affff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:07.971 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.991+0000 7f23affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:07.971 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.991+0000 7f23affff640 10 monclient(hunting): my global_id is 4395 2024-03-29T11:18:07.971 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.991+0000 7f23affff640 1 -- 172.21.15.47:0/867693398 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23c80ac7b0 con 0x7f23c81457f0 2024-03-29T11:18:07.971 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.991+0000 7f23affff640 1 -- 172.21.15.47:0/867693398 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2559950322 0 0) 0x7f23b4004eb0 con 0x7f23c80a9d70 2024-03-29T11:18:07.971 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.991+0000 7f23affff640 1 -- 172.21.15.47:0/867693398 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23c8137b50 con 0x7f23c80a9d70 2024-03-29T11:18:07.971 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.991+0000 7f23affff640 1 -- 172.21.15.47:0/867693398 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f23bc0018c0 con 0x7f23c80a9d70 2024-03-29T11:18:07.971 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.992+0000 7f23affff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:07.971 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.992+0000 7f23affff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:18:07.971 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.992+0000 7f23affff640 10 monclient(hunting): dump: 2024-03-29T11:18:07.971 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:18:07.971 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:18:07.972 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:07.972 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:07.972 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:07.972 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:07.972 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:18:07.972 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:18:07.972 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:18:07.972 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:07.972 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.992+0000 7f23affff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:07.972 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.992+0000 7f23affff640 1 -- 172.21.15.47:0/867693398 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2638679792 0 0) 0x7f23c80ac7b0 con 0x7f23c81457f0 2024-03-29T11:18:07.972 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.992+0000 7f23affff640 1 -- 172.21.15.47:0/867693398 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23b4004eb0 con 0x7f23c81457f0 2024-03-29T11:18:07.972 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.992+0000 7f23affff640 1 -- 172.21.15.47:0/867693398 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f23c0001cb0 con 0x7f23c81457f0 2024-03-29T11:18:07.972 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.992+0000 7f23affff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:07.973 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.992+0000 7f23affff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:18:07.973 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.992+0000 7f23affff640 10 monclient(hunting): dump: 2024-03-29T11:18:07.973 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:18:07.973 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:18:07.973 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:07.973 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:07.973 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:07.973 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:07.973 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:18:07.974 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:18:07.974 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:18:07.974 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:07.974 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.992+0000 7f23affff640 1 -- 172.21.15.47:0/867693398 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2077730823 0 0) 0x7f23c8137b50 con 0x7f23c80a9d70 2024-03-29T11:18:07.974 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.992+0000 7f23affff640 1 -- 172.21.15.47:0/867693398 >> v1:172.21.15.121:6789/0 conn(0x7f23c813f650 legacy=0x7f23c8136d30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:18:07.976 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.996+0000 7f23affff640 1 -- 172.21.15.47:0/867693398 >> v1:172.21.15.47:6789/0 conn(0x7f23c81457f0 legacy=0x7f23c8137440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:07.976 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.996+0000 7f23affff640 10 monclient: _finish_hunting 0 2024-03-29T11:18:07.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.996+0000 7f23affff640 1 monclient: found mon.c 2024-03-29T11:18:07.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.996+0000 7f23affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:07.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.996+0000 7f23affff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:18:07.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.996+0000 7f23affff640 1 -- 172.21.15.47:0/867693398 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23c8144c80 con 0x7f23c80a9d70 2024-03-29T11:18:07.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.996+0000 7f23affff640 10 monclient: _finish_auth 0 2024-03-29T11:18:07.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.996+0000 7f23affff640 10 monclient: _check_auth_tickets 2024-03-29T11:18:07.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.996+0000 7f23affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:07.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.996+0000 7f23cd89d640 5 monclient: authenticate success, global_id 4445 2024-03-29T11:18:07.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.996+0000 7f23cd89d640 10 monclient: _renew_subs 2024-03-29T11:18:07.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.996+0000 7f23cd89d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:18:07.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.996+0000 7f23cd89d640 1 -- 172.21.15.47:0/867693398 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23c8142d00 con 0x7f23c80a9d70 2024-03-29T11:18:07.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.996+0000 7f23cd89d640 10 client.4445.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:07.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.996+0000 7f23cd89d640 10 monclient: _renew_subs 2024-03-29T11:18:07.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.996+0000 7f23cd89d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:18:07.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.996+0000 7f23cd89d640 1 -- 172.21.15.47:0/867693398 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f23c813ff80 con 0x7f23c80a9d70 2024-03-29T11:18:07.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.996+0000 7f23cd89d640 1 librados: init done 2024-03-29T11:18:07.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.998+0000 7f23affff640 1 -- 172.21.15.47:0/867693398 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23bc003610 con 0x7f23c80a9d70 2024-03-29T11:18:07.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.998+0000 7f23affff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:07.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.998+0000 7f23affff640 1 -- 172.21.15.47:0/867693398 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f23bc003960 con 0x7f23c80a9d70 2024-03-29T11:18:07.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.998+0000 7f23affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:07.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.998+0000 7f23affff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:07.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.998+0000 7f23affff640 10 monclient: dump: 2024-03-29T11:18:07.978 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:18:07.979 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:18:07.979 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:07.979 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:07.979 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:07.979 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:07.979 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:18:07.979 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:18:07.979 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:18:07.979 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:07.979 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:07.998+0000 7f23c57fa640 4 set_mon_vals no callback set 2024-03-29T11:18:07.980 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:08.001+0000 7f23cd89d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:07.980 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:08.001+0000 7f23affff640 1 -- 172.21.15.47:0/867693398 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7f23bc003ac0 con 0x7f23c80a9d70 2024-03-29T11:18:07.980 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:08.002+0000 7f23affff640 1 -- 172.21.15.47:0/867693398 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7f23bc0849a0 con 0x7f23c80a9d70 2024-03-29T11:18:07.980 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:08.002+0000 7f23affff640 10 client.4445.objecter ms_dispatch 0x7f23c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:18:07.980 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:08.002+0000 7f23affff640 3 client.4445.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:18:07.980 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:08.002+0000 7f23affff640 3 client.4445.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:18:07.981 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:08.002+0000 7f23affff640 20 client.4445.objecter dump_active .. 0 homeless 2024-03-29T11:18:07.981 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:08.001+0000 7f23cd89d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:07.981 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:08.002+0000 7f23cd89d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:18:07.981 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:08.002+0000 7f23cd89d640 1 -- 172.21.15.47:0/867693398 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23a8005080 con 0x7f23c80a9d70 2024-03-29T11:18:07.981 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:08.002+0000 7f23c67fc640 10 monclient: get_auth_request con 0x7f23b406de90 auth_method 0 2024-03-29T11:18:07.981 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:08.004+0000 7f23affff640 10 client.4445.objecter ms_handle_connect 0x7f23b406de90 2024-03-29T11:18:07.991 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:08.014+0000 7f23affff640 1 -- 172.21.15.47:0/867693398 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f23bc050280 con 0x7f23c80a9d70 2024-03-29T11:18:07.991 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:08.014+0000 7f23affff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:07.991 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:08.014+0000 7f23affff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:08.393 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:08.416+0000 7f23cd89d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:08.393 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:08.416+0000 7f23cd89d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:08.393 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:08.416+0000 7f23cd89d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:18:08.393 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:08.416+0000 7f23cd89d640 1 -- 172.21.15.47:0/867693398 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23a8005330 con 0x7f23c80a9d70 2024-03-29T11:18:08.395 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:08.418+0000 7f23affff640 1 -- 172.21.15.47:0/867693398 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7f23bc0521b0 con 0x7f23c80a9d70 2024-03-29T11:18:08.395 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:08.418+0000 7f23affff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:08.395 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:08.418+0000 7f23affff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:08.396 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:18:08.396 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:18:08.401 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:08.423+0000 7f23cd89d640 10 librados: watch_flush enter 2024-03-29T11:18:08.401 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:08.423+0000 7f23cd89d640 10 librados: watch_flush exit 2024-03-29T11:18:08.401 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:08.423+0000 7f23cd89d640 20 client.4445.objecter shutdown clearing up homeless session... 2024-03-29T11:18:08.401 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:08.423+0000 7f23cd89d640 10 client.4445.objecter successfully canceled tick 2024-03-29T11:18:08.401 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:08.423+0000 7f23cd89d640 1 -- 172.21.15.47:0/867693398 >> v1:172.21.15.121:6816/1964117370 conn(0x7f23b406de90 legacy=0x7f23b4070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:08.401 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:08.423+0000 7f23cd89d640 10 monclient: shutdown 2024-03-29T11:18:08.401 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:08.423+0000 7f23cd89d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:08.401 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:08.423+0000 7f23cd89d640 1 -- 172.21.15.47:0/867693398 >> v1:172.21.15.47:6790/0 conn(0x7f23c80a9d70 legacy=0x7f23c81425f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:08.402 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:08.423+0000 7f23cd89d640 1 -- 172.21.15.47:0/867693398 shutdown_connections 2024-03-29T11:18:08.402 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:08.423+0000 7f23cd89d640 1 -- 172.21.15.47:0/867693398 >> 172.21.15.47:0/867693398 conn(0x7f23c8066a00 msgr2=0x7f23c8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:08.403 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:08.426+0000 7f23cd89d640 1 -- 172.21.15.47:0/867693398 shutdown_connections 2024-03-29T11:18:08.407 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:08.431+0000 7f23cd89d640 1 -- 172.21.15.47:0/867693398 wait complete. 2024-03-29T11:18:08.409 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:08.433+0000 7f23cd89d640 1 librados: shutdown 2024-03-29T11:18:08.429 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:18:15.236 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:18:15.408 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.430+0000 7f2230f9d640 10 monclient: get_monmap_and_config 2024-03-29T11:18:15.408 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.431+0000 7f2230f9d640 10 monclient: build_initial_monmap 2024-03-29T11:18:15.408 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.431+0000 7f2230f9d640 10 monclient: monmap: 2024-03-29T11:18:15.408 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:18:15.408 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:18:15.408 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:18:15.431980+0000 2024-03-29T11:18:15.408 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:18:15.431980+0000 2024-03-29T11:18:15.409 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:15.409 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:15.409 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:18:15.409 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:18:15.409 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:18:15.409 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:15.409 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.432+0000 7f2230f9d640 1 Processor -- start 2024-03-29T11:18:15.409 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.432+0000 7f2230f9d640 1 -- start start 2024-03-29T11:18:15.410 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.432+0000 7f2230f9d640 10 monclient: init 2024-03-29T11:18:15.411 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.433+0000 7f2230f9d640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:15.411 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.433+0000 7f2230f9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:18:15.411 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.434+0000 7f2230f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f222c070120 addr v1:172.21.15.47:6790/0 2024-03-29T11:18:15.411 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.434+0000 7f2230f9d640 10 monclient(hunting): picked mon.noname-a con 0x7f222c0b0420 addr v1:172.21.15.47:6789/0 2024-03-29T11:18:15.411 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.434+0000 7f2230f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f222c070500 addr v1:172.21.15.121:6789/0 2024-03-29T11:18:15.411 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.434+0000 7f2230f9d640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f222c072090 con 0x7f222c0b0420 2024-03-29T11:18:15.411 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.434+0000 7f2230f9d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f222c072370 con 0x7f222c070500 2024-03-29T11:18:15.412 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.434+0000 7f2230f9d640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f222c072650 con 0x7f222c070120 2024-03-29T11:18:15.412 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.434+0000 7f2230f9d640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:15.412 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.434+0000 7f2230f9d640 10 monclient(hunting): authenticate will time out at 1138.622039s 2024-03-29T11:18:15.412 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.434+0000 7f222ad9d640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f222c070500 0x7f222c14b810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.47:51924/0 (socket says 172.21.15.47:51924) 2024-03-29T11:18:15.412 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.434+0000 7f222ad9d640 1 -- 172.21.15.47:0/2927489691 learned_addr learned my addr 172.21.15.47:0/2927489691 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:18:15.412 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.435+0000 7f222959a640 1 -- 172.21.15.47:0/2927489691 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2101683380 0 0) 0x7f222c072370 con 0x7f222c070500 2024-03-29T11:18:15.412 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.435+0000 7f222959a640 10 monclient(hunting): _negotiate 2024-03-29T11:18:15.413 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.435+0000 7f222959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:15.413 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.435+0000 7f222959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:15.413 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.435+0000 7f222959a640 10 monclient(hunting): my global_id is 4405 2024-03-29T11:18:15.413 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.435+0000 7f222959a640 1 -- 172.21.15.47:0/2927489691 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2214003520 con 0x7f222c070500 2024-03-29T11:18:15.413 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.436+0000 7f222959a640 1 -- 172.21.15.47:0/2927489691 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 791573184 0 0) 0x7f2214003520 con 0x7f222c070500 2024-03-29T11:18:15.413 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.436+0000 7f222959a640 1 -- 172.21.15.47:0/2927489691 >> v1:172.21.15.47:6790/0 conn(0x7f222c070120 legacy=0x7f222c13d350 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:18:15.414 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.437+0000 7f222959a640 1 -- 172.21.15.47:0/2927489691 >> v1:172.21.15.47:6789/0 conn(0x7f222c0b0420 legacy=0x7f222c1405b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:15.414 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.437+0000 7f222959a640 10 monclient: _finish_hunting 0 2024-03-29T11:18:15.414 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.437+0000 7f222959a640 1 monclient: found mon.noname-c 2024-03-29T11:18:15.414 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.437+0000 7f222959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:15.414 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.437+0000 7f222959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:18:15.414 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.437+0000 7f222959a640 1 -- 172.21.15.47:0/2927489691 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f222c140090 con 0x7f222c070500 2024-03-29T11:18:15.414 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.437+0000 7f222959a640 10 monclient: _finish_auth 0 2024-03-29T11:18:15.414 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.437+0000 7f222959a640 10 monclient: _check_auth_tickets 2024-03-29T11:18:15.415 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.437+0000 7f222959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:15.415 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.437+0000 7f222959a640 1 -- 172.21.15.47:0/2927489691 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f22240019f0 con 0x7f222c070500 2024-03-29T11:18:15.415 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.437+0000 7f222959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:15.415 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.437+0000 7f222959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:18:15.415 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.437+0000 7f222959a640 10 monclient: dump: 2024-03-29T11:18:15.415 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:18:15.415 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:18:15.415 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:15.415 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:15.415 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:15.415 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:15.415 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:18:15.415 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:18:15.415 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:18:15.415 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:15.416 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.437+0000 7f2230f9d640 5 monclient: authenticate success, global_id 4405 2024-03-29T11:18:15.416 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.437+0000 7f2230f9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:15.416 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.438+0000 7f222959a640 1 -- 172.21.15.47:0/2927489691 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2224001c10 con 0x7f222c070500 2024-03-29T11:18:15.416 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.438+0000 7f222959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:15.416 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.438+0000 7f2230f9d640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:15.417 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.438+0000 7f2230f9d640 4 set_mon_vals no callback set 2024-03-29T11:18:15.417 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.438+0000 7f2230f9d640 10 monclient: shutdown 2024-03-29T11:18:15.417 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.439+0000 7f2230f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:15.417 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.439+0000 7f2230f9d640 1 -- 172.21.15.47:0/2927489691 >> v1:172.21.15.121:6789/0 conn(0x7f222c070500 legacy=0x7f222c14b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:15.417 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.439+0000 7f222959a640 1 -- 172.21.15.47:0/2927489691 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f2224001fe0 con 0x7f222c070500 2024-03-29T11:18:15.417 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.439+0000 7f222959a640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:18:15.417 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.439+0000 7f2230f9d640 1 -- 172.21.15.47:0/2927489691 shutdown_connections 2024-03-29T11:18:15.417 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.439+0000 7f2230f9d640 1 -- 172.21.15.47:0/2927489691 >> 172.21.15.47:0/2927489691 conn(0x7f222c067500 msgr2=0x7f222c0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:15.417 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.439+0000 7f2230f9d640 1 -- 172.21.15.47:0/2927489691 shutdown_connections 2024-03-29T11:18:15.417 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.439+0000 7f2230f9d640 1 -- 172.21.15.47:0/2927489691 wait complete. 2024-03-29T11:18:15.419 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.440+0000 7f2230f9d640 10 monclient: build_initial_monmap 2024-03-29T11:18:15.419 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.440+0000 7f2230f9d640 10 monclient: monmap: 2024-03-29T11:18:15.419 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:18:15.419 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:15.419 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:18:15.419 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:18:15.419 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:15.419 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:15.419 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:18:15.419 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:18:15.419 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:18:15.419 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:15.419 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.441+0000 7f2230f9d640 1 librados: starting msgr at 2024-03-29T11:18:15.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.441+0000 7f2230f9d640 1 librados: starting objecter 2024-03-29T11:18:15.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.441+0000 7f2230f9d640 1 Processor -- start 2024-03-29T11:18:15.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.441+0000 7f2230f9d640 1 -- start start 2024-03-29T11:18:15.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.441+0000 7f2230f9d640 1 librados: setting wanted keys 2024-03-29T11:18:15.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.441+0000 7f2230f9d640 1 librados: calling monclient init 2024-03-29T11:18:15.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.441+0000 7f2230f9d640 10 monclient: init 2024-03-29T11:18:15.425 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.442+0000 7f2230f9d640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:15.425 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.442+0000 7f2230f9d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:18:15.425 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.442+0000 7f2230f9d640 10 monclient(hunting): picked mon.noname-a con 0x7f222c070120 addr v1:172.21.15.47:6789/0 2024-03-29T11:18:15.425 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.442+0000 7f2230f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f222c070500 addr v1:172.21.15.121:6789/0 2024-03-29T11:18:15.425 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.443+0000 7f2230f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f222c0b0420 addr v1:172.21.15.47:6790/0 2024-03-29T11:18:15.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.444+0000 7f222a59c640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f222c070120 0x7f222c138610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:38870/0 (socket says 172.21.15.47:38870) 2024-03-29T11:18:15.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.444+0000 7f222a59c640 1 -- 172.21.15.47:0/1411028464 learned_addr learned my addr 172.21.15.47:0/1411028464 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:18:15.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.444+0000 7f2229d9b640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f222c070500 0x7f222c0ad3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.47:58540/0 (socket says 172.21.15.47:58540) 2024-03-29T11:18:15.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.444+0000 7f2230f9d640 1 -- 172.21.15.47:0/1411028464 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f222c138d20 con 0x7f222c070120 2024-03-29T11:18:15.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.444+0000 7f221affd640 10 client.?.objecter ms_handle_connect 0x7f222c070120 2024-03-29T11:18:15.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.444+0000 7f221affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:15.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.445+0000 7f221affd640 10 client.?.objecter ms_handle_connect 0x7f222c070500 2024-03-29T11:18:15.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.444+0000 7f2230f9d640 1 -- 172.21.15.47:0/1411028464 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f222c0aca80 con 0x7f222c070500 2024-03-29T11:18:15.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.445+0000 7f2230f9d640 1 -- 172.21.15.47:0/1411028464 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f222c0af360 con 0x7f222c0b0420 2024-03-29T11:18:15.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.445+0000 7f221affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:15.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.445+0000 7f221affd640 1 -- 172.21.15.47:0/1411028464 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1494219522 0 0) 0x7f222c138d20 con 0x7f222c070120 2024-03-29T11:18:15.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.445+0000 7f2230f9d640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:15.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.445+0000 7f2230f9d640 10 monclient(hunting): authenticate will time out at 1138.633379s 2024-03-29T11:18:15.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.445+0000 7f221affd640 10 monclient(hunting): _negotiate 2024-03-29T11:18:15.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.445+0000 7f221affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:15.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.445+0000 7f221affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:15.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.445+0000 7f221affd640 10 monclient(hunting): my global_id is 4398 2024-03-29T11:18:15.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.445+0000 7f221affd640 1 -- 172.21.15.47:0/1411028464 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f220c004eb0 con 0x7f222c070120 2024-03-29T11:18:15.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.445+0000 7f221affd640 1 -- 172.21.15.47:0/1411028464 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1742630764 0 0) 0x7f222c0aca80 con 0x7f222c070500 2024-03-29T11:18:15.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.445+0000 7f221affd640 10 monclient(hunting): _negotiate 2024-03-29T11:18:15.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.445+0000 7f221affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:15.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.446+0000 7f221affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:15.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.446+0000 7f221affd640 10 monclient(hunting): my global_id is 4408 2024-03-29T11:18:15.428 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.446+0000 7f221affd640 1 -- 172.21.15.47:0/1411028464 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f222c138d20 con 0x7f222c070500 2024-03-29T11:18:15.428 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.446+0000 7f221affd640 1 -- 172.21.15.47:0/1411028464 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 548641025 0 0) 0x7f220c004eb0 con 0x7f222c070120 2024-03-29T11:18:15.428 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.446+0000 7f221affd640 1 -- 172.21.15.47:0/1411028464 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f222c0aca80 con 0x7f222c070120 2024-03-29T11:18:15.428 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.447+0000 7f221affd640 1 -- 172.21.15.47:0/1411028464 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f221c001960 con 0x7f222c070120 2024-03-29T11:18:15.428 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.447+0000 7f221affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:15.428 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.447+0000 7f221affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:18:15.428 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.447+0000 7f221affd640 10 monclient(hunting): dump: 2024-03-29T11:18:15.428 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:18:15.428 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:18:15.428 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:15.428 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:15.428 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:15.429 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:15.429 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:18:15.429 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:18:15.429 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:18:15.429 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:15.429 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.447+0000 7f221affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:15.429 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.447+0000 7f221affd640 10 client.?.objecter ms_handle_connect 0x7f222c0b0420 2024-03-29T11:18:15.429 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.447+0000 7f221affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:15.429 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.447+0000 7f221affd640 1 -- 172.21.15.47:0/1411028464 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 92807421 0 0) 0x7f222c138d20 con 0x7f222c070500 2024-03-29T11:18:15.429 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.447+0000 7f221affd640 1 -- 172.21.15.47:0/1411028464 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f220c004eb0 con 0x7f222c070500 2024-03-29T11:18:15.429 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.447+0000 7f221affd640 1 -- 172.21.15.47:0/1411028464 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f2220001790 con 0x7f222c070500 2024-03-29T11:18:15.429 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.447+0000 7f221affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:15.429 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.447+0000 7f221affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:18:15.429 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.447+0000 7f221affd640 10 monclient(hunting): dump: 2024-03-29T11:18:15.430 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:18:15.430 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:18:15.430 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:15.430 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:15.430 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:15.430 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:15.430 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:18:15.430 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:18:15.430 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:18:15.431 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:15.431 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.447+0000 7f221affd640 1 -- 172.21.15.47:0/1411028464 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 862226702 0 0) 0x7f222c0af360 con 0x7f222c0b0420 2024-03-29T11:18:15.431 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.447+0000 7f221affd640 10 monclient(hunting): _negotiate 2024-03-29T11:18:15.431 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.447+0000 7f221affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:15.431 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.447+0000 7f221affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:15.431 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.447+0000 7f221affd640 10 monclient(hunting): my global_id is 4448 2024-03-29T11:18:15.431 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.447+0000 7f221affd640 1 -- 172.21.15.47:0/1411028464 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f222c138d20 con 0x7f222c0b0420 2024-03-29T11:18:15.431 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.447+0000 7f221affd640 1 -- 172.21.15.47:0/1411028464 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4153920672 0 0) 0x7f222c138d20 con 0x7f222c0b0420 2024-03-29T11:18:15.431 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.447+0000 7f221affd640 1 -- 172.21.15.47:0/1411028464 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f222c0af360 con 0x7f222c0b0420 2024-03-29T11:18:15.431 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.447+0000 7f221affd640 1 -- 172.21.15.47:0/1411028464 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f2224001f20 con 0x7f222c0b0420 2024-03-29T11:18:15.431 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.448+0000 7f221affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:15.431 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.448+0000 7f221affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:15.431 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.448+0000 7f221affd640 10 monclient(hunting): dump: 2024-03-29T11:18:15.431 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:18:15.431 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:18:15.432 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:15.432 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:15.432 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:15.432 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:15.432 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:18:15.432 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:18:15.432 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:18:15.432 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:15.432 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.448+0000 7f221affd640 1 -- 172.21.15.47:0/1411028464 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4098830800 0 0) 0x7f222c0aca80 con 0x7f222c070120 2024-03-29T11:18:15.432 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.448+0000 7f221affd640 1 -- 172.21.15.47:0/1411028464 >> v1:172.21.15.47:6790/0 conn(0x7f222c0b0420 legacy=0x7f222c1dbb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:15.432 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.448+0000 7f221affd640 1 -- 172.21.15.47:0/1411028464 >> v1:172.21.15.121:6789/0 conn(0x7f222c070500 legacy=0x7f222c0ad3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:15.432 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.448+0000 7f221affd640 10 monclient: _finish_hunting 0 2024-03-29T11:18:15.432 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.448+0000 7f221affd640 1 monclient: found mon.a 2024-03-29T11:18:15.433 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.448+0000 7f221affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:15.433 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.448+0000 7f221affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:18:15.433 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.448+0000 7f221affd640 1 -- 172.21.15.47:0/1411028464 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f222c0af4f0 con 0x7f222c070120 2024-03-29T11:18:15.433 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.448+0000 7f221affd640 10 monclient: _finish_auth 0 2024-03-29T11:18:15.433 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.448+0000 7f221affd640 10 monclient: _check_auth_tickets 2024-03-29T11:18:15.433 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.448+0000 7f221affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:15.433 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.448+0000 7f2230f9d640 5 monclient: authenticate success, global_id 4398 2024-03-29T11:18:15.433 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.448+0000 7f2230f9d640 10 monclient: _renew_subs 2024-03-29T11:18:15.433 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.448+0000 7f2230f9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:18:15.433 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.448+0000 7f2230f9d640 1 -- 172.21.15.47:0/1411028464 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f222c0af7a0 con 0x7f222c070120 2024-03-29T11:18:15.435 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.448+0000 7f2230f9d640 10 client.4398.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:15.435 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.448+0000 7f2230f9d640 10 monclient: _renew_subs 2024-03-29T11:18:15.435 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.448+0000 7f2230f9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:18:15.435 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.448+0000 7f2230f9d640 1 -- 172.21.15.47:0/1411028464 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f222c0adc90 con 0x7f222c070120 2024-03-29T11:18:15.435 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.448+0000 7f2230f9d640 1 librados: init done 2024-03-29T11:18:15.436 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.449+0000 7f221affd640 1 -- 172.21.15.47:0/1411028464 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f221c0035f0 con 0x7f222c070120 2024-03-29T11:18:15.436 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.449+0000 7f221affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:15.436 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.449+0000 7f221affd640 1 -- 172.21.15.47:0/1411028464 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f221c003940 con 0x7f222c070120 2024-03-29T11:18:15.436 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.449+0000 7f221affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:15.436 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.449+0000 7f221affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:18:15.436 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.449+0000 7f221affd640 10 monclient: dump: 2024-03-29T11:18:15.436 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:18:15.436 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:18:15.436 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:15.436 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:15.436 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:15.436 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:15.436 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:18:15.436 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:18:15.436 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:18:15.437 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:15.437 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.449+0000 7f2228d99640 4 set_mon_vals no callback set 2024-03-29T11:18:15.437 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.451+0000 7f221affd640 1 -- 172.21.15.47:0/1411028464 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7f221c003aa0 con 0x7f222c070120 2024-03-29T11:18:15.437 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.451+0000 7f2230f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:15.437 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.451+0000 7f2230f9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:15.437 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.451+0000 7f2230f9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:18:15.437 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.451+0000 7f2230f9d640 1 -- 172.21.15.47:0/1411028464 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2210005080 con 0x7f222c070120 2024-03-29T11:18:15.437 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.452+0000 7f221affd640 1 -- 172.21.15.47:0/1411028464 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7f221c084a30 con 0x7f222c070120 2024-03-29T11:18:15.437 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.452+0000 7f221affd640 10 client.4398.objecter ms_dispatch 0x7f222c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:18:15.437 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.452+0000 7f221affd640 3 client.4398.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:18:15.437 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.452+0000 7f221affd640 3 client.4398.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:18:15.437 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.452+0000 7f221affd640 20 client.4398.objecter dump_active .. 0 homeless 2024-03-29T11:18:15.437 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.452+0000 7f2229d9b640 10 monclient: get_auth_request con 0x7f220c06e0f0 auth_method 0 2024-03-29T11:18:15.437 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.452+0000 7f221affd640 10 client.4398.objecter ms_handle_connect 0x7f220c06e0f0 2024-03-29T11:18:15.438 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.461+0000 7f221affd640 1 -- 172.21.15.47:0/1411028464 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f221c050310 con 0x7f222c070120 2024-03-29T11:18:15.438 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.461+0000 7f221affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:15.438 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.461+0000 7f221affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:15.768 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.791+0000 7f2230f9d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:15.768 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.791+0000 7f2230f9d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:15.768 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.791+0000 7f2230f9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:18:15.769 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.791+0000 7f2230f9d640 1 -- 172.21.15.47:0/1411028464 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2210005330 con 0x7f222c070120 2024-03-29T11:18:15.770 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.794+0000 7f221affd640 1 -- 172.21.15.47:0/1411028464 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7f221c052240 con 0x7f222c070120 2024-03-29T11:18:15.771 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.794+0000 7f221affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:15.771 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.794+0000 7f221affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:15.774 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:18:15.774 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:18:15.780 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.800+0000 7f2230f9d640 10 librados: watch_flush enter 2024-03-29T11:18:15.780 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.802+0000 7f2230f9d640 10 librados: watch_flush exit 2024-03-29T11:18:15.780 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.802+0000 7f2230f9d640 20 client.4398.objecter shutdown clearing up homeless session... 2024-03-29T11:18:15.780 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.802+0000 7f2230f9d640 10 client.4398.objecter successfully canceled tick 2024-03-29T11:18:15.780 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.802+0000 7f2230f9d640 1 -- 172.21.15.47:0/1411028464 >> v1:172.21.15.121:6816/1964117370 conn(0x7f220c06e0f0 legacy=0x7f220c0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:15.780 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.802+0000 7f2230f9d640 10 monclient: shutdown 2024-03-29T11:18:15.780 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.802+0000 7f2230f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:15.780 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.802+0000 7f2230f9d640 1 -- 172.21.15.47:0/1411028464 >> v1:172.21.15.47:6789/0 conn(0x7f222c070120 legacy=0x7f222c138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:15.780 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.802+0000 7f2230f9d640 1 -- 172.21.15.47:0/1411028464 shutdown_connections 2024-03-29T11:18:15.780 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.802+0000 7f2230f9d640 1 -- 172.21.15.47:0/1411028464 >> 172.21.15.47:0/1411028464 conn(0x7f222c067500 msgr2=0x7f222c1405b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:15.781 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.803+0000 7f2230f9d640 1 -- 172.21.15.47:0/1411028464 shutdown_connections 2024-03-29T11:18:15.781 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.803+0000 7f2230f9d640 1 -- 172.21.15.47:0/1411028464 wait complete. 2024-03-29T11:18:15.781 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:15.804+0000 7f2230f9d640 1 librados: shutdown 2024-03-29T11:18:15.799 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:18:22.606 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:18:22.753 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.776+0000 7f8ac8463640 10 monclient: get_monmap_and_config 2024-03-29T11:18:22.753 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.776+0000 7f8ac8463640 10 monclient: build_initial_monmap 2024-03-29T11:18:22.753 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.776+0000 7f8ac8463640 10 monclient: monmap: 2024-03-29T11:18:22.753 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:18:22.754 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:18:22.754 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:18:22.777203+0000 2024-03-29T11:18:22.754 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:18:22.777203+0000 2024-03-29T11:18:22.754 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:22.754 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:22.754 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:18:22.754 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:18:22.754 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:18:22.754 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:22.755 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.777+0000 7f8ac8463640 1 Processor -- start 2024-03-29T11:18:22.755 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.777+0000 7f8ac8463640 1 -- start start 2024-03-29T11:18:22.755 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.777+0000 7f8ac8463640 10 monclient: init 2024-03-29T11:18:22.756 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.778+0000 7f8ac8463640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:22.756 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.778+0000 7f8ac8463640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:18:22.756 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.778+0000 7f8ac8463640 10 monclient(hunting): picked mon.noname-a con 0x7f8ac01457f0 addr v1:172.21.15.47:6789/0 2024-03-29T11:18:22.756 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.778+0000 7f8ac8463640 10 monclient(hunting): picked mon.noname-b con 0x7f8ac013f650 addr v1:172.21.15.47:6790/0 2024-03-29T11:18:22.756 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.778+0000 7f8ac8463640 10 monclient(hunting): picked mon.noname-c con 0x7f8ac00a9d70 addr v1:172.21.15.121:6789/0 2024-03-29T11:18:22.756 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.778+0000 7f8ac8463640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ac00aaef0 con 0x7f8ac01457f0 2024-03-29T11:18:22.756 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.778+0000 7f8ac8463640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ac0145bd0 con 0x7f8ac00a9d70 2024-03-29T11:18:22.756 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.778+0000 7f8ac8463640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ac0145eb0 con 0x7f8ac013f650 2024-03-29T11:18:22.756 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.778+0000 7f8ac8463640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:22.756 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.778+0000 7f8ac8463640 10 monclient(hunting): authenticate will time out at 1145.966680s 2024-03-29T11:18:22.756 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.779+0000 7f8ac61ff640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f8ac01457f0 0x7f8ac013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:52628/0 (socket says 172.21.15.47:52628) 2024-03-29T11:18:22.756 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.779+0000 7f8ac61ff640 1 -- 172.21.15.47:0/4061219940 learned_addr learned my addr 172.21.15.47:0/4061219940 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:18:22.758 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.779+0000 7f8ac51fd640 1 -- 172.21.15.47:0/4061219940 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3217286204 0 0) 0x7f8ac00aaef0 con 0x7f8ac01457f0 2024-03-29T11:18:22.759 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.779+0000 7f8ac51fd640 10 monclient(hunting): _negotiate 2024-03-29T11:18:22.759 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.779+0000 7f8ac51fd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:22.759 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.779+0000 7f8ac51fd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:22.759 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.779+0000 7f8ac51fd640 10 monclient(hunting): my global_id is 4401 2024-03-29T11:18:22.759 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.779+0000 7f8ac51fd640 1 -- 172.21.15.47:0/4061219940 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8aac003520 con 0x7f8ac01457f0 2024-03-29T11:18:22.759 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.779+0000 7f8ac51fd640 1 -- 172.21.15.47:0/4061219940 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1622609998 0 0) 0x7f8aac003520 con 0x7f8ac01457f0 2024-03-29T11:18:22.759 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.779+0000 7f8ac51fd640 1 -- 172.21.15.47:0/4061219940 >> v1:172.21.15.47:6790/0 conn(0x7f8ac013f650 legacy=0x7f8ac00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:22.759 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.779+0000 7f8ac51fd640 1 -- 172.21.15.47:0/4061219940 >> v1:172.21.15.121:6789/0 conn(0x7f8ac00a9d70 legacy=0x7f8ac014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:22.759 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.779+0000 7f8ac51fd640 10 monclient: _finish_hunting 0 2024-03-29T11:18:22.759 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.779+0000 7f8ac51fd640 1 monclient: found mon.noname-a 2024-03-29T11:18:22.759 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.779+0000 7f8ac51fd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:22.759 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.779+0000 7f8ac51fd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-29T11:18:22.759 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.780+0000 7f8ac51fd640 1 -- 172.21.15.47:0/4061219940 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ac0146340 con 0x7f8ac01457f0 2024-03-29T11:18:22.760 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.780+0000 7f8ac51fd640 10 monclient: _finish_auth 0 2024-03-29T11:18:22.760 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.780+0000 7f8ac51fd640 10 monclient: _check_auth_tickets 2024-03-29T11:18:22.760 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.780+0000 7f8ac51fd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:22.760 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.780+0000 7f8ac51fd640 1 -- 172.21.15.47:0/4061219940 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f8abc001690 con 0x7f8ac01457f0 2024-03-29T11:18:22.760 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.780+0000 7f8ac51fd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:22.760 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.780+0000 7f8ac51fd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:18:22.760 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.780+0000 7f8ac51fd640 10 monclient: dump: 2024-03-29T11:18:22.760 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:18:22.760 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:18:22.760 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:22.760 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:22.760 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:22.760 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:22.760 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:18:22.760 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:18:22.761 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:18:22.761 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:22.761 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.780+0000 7f8ac8463640 5 monclient: authenticate success, global_id 4401 2024-03-29T11:18:22.761 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.780+0000 7f8ac8463640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:22.761 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.780+0000 7f8ac51fd640 1 -- 172.21.15.47:0/4061219940 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8abc001870 con 0x7f8ac01457f0 2024-03-29T11:18:22.761 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.780+0000 7f8ac51fd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:22.761 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.780+0000 7f8ac8463640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:22.761 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.780+0000 7f8ac8463640 4 set_mon_vals no callback set 2024-03-29T11:18:22.761 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.781+0000 7f8ac51fd640 1 -- 172.21.15.47:0/4061219940 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f8abc001c80 con 0x7f8ac01457f0 2024-03-29T11:18:22.761 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.781+0000 7f8ac51fd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:22.761 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.781+0000 7f8ac51fd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:18:22.761 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.781+0000 7f8ac51fd640 10 monclient: dump: 2024-03-29T11:18:22.761 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:18:22.762 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:18:22.762 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:22.762 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:22.762 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:22.762 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:22.762 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:18:22.762 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:18:22.762 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:18:22.762 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:22.762 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.781+0000 7f8ac8463640 10 monclient: shutdown 2024-03-29T11:18:22.762 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.781+0000 7f8ac8463640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:22.762 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.781+0000 7f8ac8463640 1 -- 172.21.15.47:0/4061219940 >> v1:172.21.15.47:6789/0 conn(0x7f8ac01457f0 legacy=0x7f8ac013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:22.769 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.782+0000 7f8ac8463640 1 -- 172.21.15.47:0/4061219940 shutdown_connections 2024-03-29T11:18:22.770 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.782+0000 7f8ac8463640 1 -- 172.21.15.47:0/4061219940 >> 172.21.15.47:0/4061219940 conn(0x7f8ac0066a00 msgr2=0x7f8ac0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:22.770 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.782+0000 7f8ac8463640 1 -- 172.21.15.47:0/4061219940 shutdown_connections 2024-03-29T11:18:22.770 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.782+0000 7f8ac8463640 1 -- 172.21.15.47:0/4061219940 wait complete. 2024-03-29T11:18:22.770 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.783+0000 7f8ac8463640 10 monclient: build_initial_monmap 2024-03-29T11:18:22.770 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.783+0000 7f8ac8463640 10 monclient: monmap: 2024-03-29T11:18:22.770 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:18:22.770 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:22.770 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:18:22.770 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:18:22.770 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:22.770 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:22.770 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:18:22.771 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:18:22.771 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:18:22.771 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:22.771 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.784+0000 7f8ac8463640 1 librados: starting msgr at 2024-03-29T11:18:22.771 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.784+0000 7f8ac8463640 1 librados: starting objecter 2024-03-29T11:18:22.771 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.784+0000 7f8ac8463640 1 Processor -- start 2024-03-29T11:18:22.771 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.784+0000 7f8ac8463640 1 -- start start 2024-03-29T11:18:22.771 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.784+0000 7f8ac8463640 1 librados: setting wanted keys 2024-03-29T11:18:22.771 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.784+0000 7f8ac8463640 1 librados: calling monclient init 2024-03-29T11:18:22.771 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.784+0000 7f8ac8463640 10 monclient: init 2024-03-29T11:18:22.771 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.785+0000 7f8ac8463640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:22.771 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.785+0000 7f8ac8463640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:18:22.771 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.785+0000 7f8ac8463640 10 monclient(hunting): picked mon.noname-c con 0x7f8ac00a9d70 addr v1:172.21.15.47:6790/0 2024-03-29T11:18:22.771 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.785+0000 7f8ac8463640 10 monclient(hunting): picked mon.noname-b con 0x7f8ac013f650 addr v1:172.21.15.121:6789/0 2024-03-29T11:18:22.771 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.785+0000 7f8ac8463640 10 monclient(hunting): picked mon.noname-a con 0x7f8ac01457f0 addr v1:172.21.15.47:6789/0 2024-03-29T11:18:22.772 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.785+0000 7f8ac8463640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ac01cf1e0 con 0x7f8ac01457f0 2024-03-29T11:18:22.772 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.785+0000 7f8ac8463640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ac01cc880 con 0x7f8ac013f650 2024-03-29T11:18:22.772 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.785+0000 7f8ac8463640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ac01ccb60 con 0x7f8ac00a9d70 2024-03-29T11:18:22.772 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.785+0000 7f8ac8463640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:22.772 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.785+0000 7f8ac8463640 10 monclient(hunting): authenticate will time out at 1145.973366s 2024-03-29T11:18:22.772 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.785+0000 7f8ac61ff640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f8ac00a9d70 0x7f8ac013b370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:60872/0 (socket says 172.21.15.47:60872) 2024-03-29T11:18:22.772 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.785+0000 7f8ac61ff640 1 -- 172.21.15.47:0/2055002196 learned_addr learned my addr 172.21.15.47:0/2055002196 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:18:22.772 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.786+0000 7f8ab6ffd640 10 client.?.objecter ms_handle_connect 0x7f8ac013f650 2024-03-29T11:18:22.772 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.786+0000 7f8ab6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:22.772 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.786+0000 7f8ab6ffd640 10 client.?.objecter ms_handle_connect 0x7f8ac01457f0 2024-03-29T11:18:22.772 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.786+0000 7f8ab6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:22.772 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.786+0000 7f8ab6ffd640 10 client.?.objecter ms_handle_connect 0x7f8ac00a9d70 2024-03-29T11:18:22.772 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.786+0000 7f8ab6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:22.772 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.786+0000 7f8ab6ffd640 1 -- 172.21.15.47:0/2055002196 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 403287757 0 0) 0x7f8ac01ccb60 con 0x7f8ac00a9d70 2024-03-29T11:18:22.772 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.786+0000 7f8ab6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:18:22.772 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.786+0000 7f8ab6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:22.773 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.786+0000 7f8ab6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:22.773 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.786+0000 7f8ab6ffd640 10 monclient(hunting): my global_id is 4451 2024-03-29T11:18:22.773 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.786+0000 7f8ab6ffd640 1 -- 172.21.15.47:0/2055002196 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8aa4004eb0 con 0x7f8ac00a9d70 2024-03-29T11:18:22.773 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.786+0000 7f8ab6ffd640 1 -- 172.21.15.47:0/2055002196 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 712606117 0 0) 0x7f8ac01cc880 con 0x7f8ac013f650 2024-03-29T11:18:22.773 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.786+0000 7f8ab6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:18:22.773 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.786+0000 7f8ab6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:22.773 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.786+0000 7f8ab6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:22.773 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.786+0000 7f8ab6ffd640 10 monclient(hunting): my global_id is 4411 2024-03-29T11:18:22.773 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.786+0000 7f8ab6ffd640 1 -- 172.21.15.47:0/2055002196 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ac01ccb60 con 0x7f8ac013f650 2024-03-29T11:18:22.773 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.786+0000 7f8ab6ffd640 1 -- 172.21.15.47:0/2055002196 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 55279046 0 0) 0x7f8ac01cf1e0 con 0x7f8ac01457f0 2024-03-29T11:18:22.773 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.786+0000 7f8ab6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:18:22.773 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.786+0000 7f8ab6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:22.773 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.786+0000 7f8ab6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:22.773 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.786+0000 7f8ab6ffd640 10 monclient(hunting): my global_id is 4404 2024-03-29T11:18:22.774 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.786+0000 7f8ab6ffd640 1 -- 172.21.15.47:0/2055002196 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8aa40053b0 con 0x7f8ac01457f0 2024-03-29T11:18:22.774 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.787+0000 7f8ab6ffd640 1 -- 172.21.15.47:0/2055002196 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2816743180 0 0) 0x7f8aa4004eb0 con 0x7f8ac00a9d70 2024-03-29T11:18:22.774 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.787+0000 7f8ab6ffd640 1 -- 172.21.15.47:0/2055002196 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ac01cf1e0 con 0x7f8ac00a9d70 2024-03-29T11:18:22.774 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.787+0000 7f8ab6ffd640 1 -- 172.21.15.47:0/2055002196 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f8abc001bc0 con 0x7f8ac00a9d70 2024-03-29T11:18:22.774 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.787+0000 7f8ab6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:22.774 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.787+0000 7f8ab6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:18:22.774 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.787+0000 7f8ab6ffd640 10 monclient(hunting): dump: 2024-03-29T11:18:22.774 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:18:22.774 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:18:22.774 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:22.774 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:22.774 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:22.774 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:22.775 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:18:22.775 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:18:22.775 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:18:22.775 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:22.775 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.787+0000 7f8ab6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:22.775 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.787+0000 7f8ab6ffd640 1 -- 172.21.15.47:0/2055002196 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3350978129 0 0) 0x7f8aa40053b0 con 0x7f8ac01457f0 2024-03-29T11:18:22.775 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.787+0000 7f8ab6ffd640 1 -- 172.21.15.47:0/2055002196 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8aa4004eb0 con 0x7f8ac01457f0 2024-03-29T11:18:22.775 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.787+0000 7f8ab6ffd640 1 -- 172.21.15.47:0/2055002196 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f8ab8001af0 con 0x7f8ac01457f0 2024-03-29T11:18:22.775 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.787+0000 7f8ab6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:22.775 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.787+0000 7f8ab6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:18:22.775 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.787+0000 7f8ab6ffd640 10 monclient(hunting): dump: 2024-03-29T11:18:22.775 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:18:22.776 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:18:22.776 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:22.776 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:22.776 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:22.776 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:22.776 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:18:22.776 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:18:22.776 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:18:22.776 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:22.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.787+0000 7f8ab6ffd640 1 -- 172.21.15.47:0/2055002196 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2323088602 0 0) 0x7f8aa4004eb0 con 0x7f8ac01457f0 2024-03-29T11:18:22.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.787+0000 7f8ab6ffd640 1 -- 172.21.15.47:0/2055002196 >> v1:172.21.15.47:6790/0 conn(0x7f8ac00a9d70 legacy=0x7f8ac013b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:22.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.787+0000 7f8ab6ffd640 1 -- 172.21.15.47:0/2055002196 >> v1:172.21.15.121:6789/0 conn(0x7f8ac013f650 legacy=0x7f8ac0138ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:22.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.787+0000 7f8ab6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:18:22.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.787+0000 7f8ab6ffd640 1 monclient: found mon.a 2024-03-29T11:18:22.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.787+0000 7f8ab6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:22.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.787+0000 7f8ab6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:18:22.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.787+0000 7f8ab6ffd640 1 -- 172.21.15.47:0/2055002196 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ac01cce40 con 0x7f8ac01457f0 2024-03-29T11:18:22.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.787+0000 7f8ab6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:18:22.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.787+0000 7f8ab6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:18:22.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.787+0000 7f8ab6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:22.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.787+0000 7f8ac8463640 5 monclient: authenticate success, global_id 4404 2024-03-29T11:18:22.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.787+0000 7f8ac8463640 10 monclient: _renew_subs 2024-03-29T11:18:22.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.787+0000 7f8ac8463640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:18:22.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.787+0000 7f8ac8463640 1 -- 172.21.15.47:0/2055002196 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ac01cb340 con 0x7f8ac01457f0 2024-03-29T11:18:22.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.787+0000 7f8ac8463640 10 client.4404.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:22.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.787+0000 7f8ac8463640 10 monclient: _renew_subs 2024-03-29T11:18:22.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.787+0000 7f8ac8463640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:18:22.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.787+0000 7f8ac8463640 1 -- 172.21.15.47:0/2055002196 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ac01cc450 con 0x7f8ac01457f0 2024-03-29T11:18:22.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.787+0000 7f8ac8463640 1 librados: init done 2024-03-29T11:18:22.778 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.788+0000 7f8ab6ffd640 1 -- 172.21.15.47:0/2055002196 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ab8001fe0 con 0x7f8ac01457f0 2024-03-29T11:18:22.778 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.788+0000 7f8ab6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:22.778 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.788+0000 7f8ac49fc640 4 set_mon_vals no callback set 2024-03-29T11:18:22.778 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.789+0000 7f8ab6ffd640 1 -- 172.21.15.47:0/2055002196 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f8ab8003b30 con 0x7f8ac01457f0 2024-03-29T11:18:22.778 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.789+0000 7f8ab6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:22.778 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.789+0000 7f8ab6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:18:22.778 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.789+0000 7f8ab6ffd640 10 monclient: dump: 2024-03-29T11:18:22.778 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:18:22.778 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:18:22.778 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:22.778 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:22.778 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:22.778 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:22.778 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:18:22.779 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:18:22.779 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:18:22.779 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:22.779 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.790+0000 7f8ac8463640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:22.779 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.790+0000 7f8ac8463640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:22.779 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.790+0000 7f8ab6ffd640 1 -- 172.21.15.47:0/2055002196 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7f8ab8003d50 con 0x7f8ac01457f0 2024-03-29T11:18:22.779 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.791+0000 7f8ab6ffd640 1 -- 172.21.15.47:0/2055002196 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7f8ab8083b50 con 0x7f8ac01457f0 2024-03-29T11:18:22.779 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.791+0000 7f8ab6ffd640 10 client.4404.objecter ms_dispatch 0x7f8ac0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:18:22.779 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.791+0000 7f8ab6ffd640 3 client.4404.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:18:22.779 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.791+0000 7f8ab6ffd640 3 client.4404.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:18:22.779 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.791+0000 7f8ab6ffd640 20 client.4404.objecter dump_active .. 0 homeless 2024-03-29T11:18:22.779 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.792+0000 7f8ac8463640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:18:22.779 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.792+0000 7f8ac8463640 1 -- 172.21.15.47:0/2055002196 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8aa8005080 con 0x7f8ac01457f0 2024-03-29T11:18:22.780 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.792+0000 7f8ac61ff640 10 monclient: get_auth_request con 0x7f8aa406e170 auth_method 0 2024-03-29T11:18:22.780 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.793+0000 7f8ab6ffd640 10 client.4404.objecter ms_handle_connect 0x7f8aa406e170 2024-03-29T11:18:22.780 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.800+0000 7f8ab6ffd640 1 -- 172.21.15.47:0/2055002196 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8ab804f430 con 0x7f8ac01457f0 2024-03-29T11:18:22.780 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.800+0000 7f8ab6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:22.780 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:22.800+0000 7f8ab6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:23.182 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:23.205+0000 7f8ac8463640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:23.183 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:23.205+0000 7f8ac8463640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:23.183 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:23.205+0000 7f8ac8463640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:18:23.183 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:23.205+0000 7f8ac8463640 1 -- 172.21.15.47:0/2055002196 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8aa8005330 con 0x7f8ac01457f0 2024-03-29T11:18:23.183 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:23.206+0000 7f8ab6ffd640 1 -- 172.21.15.47:0/2055002196 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7f8ab8051360 con 0x7f8ac01457f0 2024-03-29T11:18:23.184 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:23.206+0000 7f8ab6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:23.184 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:23.206+0000 7f8ab6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:23.184 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:18:23.184 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:18:23.186 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:23.209+0000 7f8ac8463640 10 librados: watch_flush enter 2024-03-29T11:18:23.186 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:23.209+0000 7f8ac8463640 10 librados: watch_flush exit 2024-03-29T11:18:23.186 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:23.209+0000 7f8ac8463640 20 client.4404.objecter shutdown clearing up homeless session... 2024-03-29T11:18:23.186 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:23.209+0000 7f8ac8463640 10 client.4404.objecter successfully canceled tick 2024-03-29T11:18:23.186 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:23.209+0000 7f8ac8463640 1 -- 172.21.15.47:0/2055002196 >> v1:172.21.15.121:6816/1964117370 conn(0x7f8aa406e170 legacy=0x7f8aa4070630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:23.187 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:23.209+0000 7f8ac8463640 10 monclient: shutdown 2024-03-29T11:18:23.187 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:23.209+0000 7f8ac8463640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:23.187 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:23.209+0000 7f8ac8463640 1 -- 172.21.15.47:0/2055002196 >> v1:172.21.15.47:6789/0 conn(0x7f8ac01457f0 legacy=0x7f8ac01e6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:23.187 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:23.209+0000 7f8ac8463640 1 -- 172.21.15.47:0/2055002196 shutdown_connections 2024-03-29T11:18:23.187 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:23.209+0000 7f8ac8463640 1 -- 172.21.15.47:0/2055002196 >> 172.21.15.47:0/2055002196 conn(0x7f8ac0066a00 msgr2=0x7f8ac0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:23.187 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:23.210+0000 7f8ac8463640 1 -- 172.21.15.47:0/2055002196 shutdown_connections 2024-03-29T11:18:23.188 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:23.210+0000 7f8ac8463640 1 -- 172.21.15.47:0/2055002196 wait complete. 2024-03-29T11:18:23.188 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:23.210+0000 7f8ac8463640 1 librados: shutdown 2024-03-29T11:18:23.202 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:18:30.009 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:18:30.156 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.179+0000 7fa41579d640 10 monclient: get_monmap_and_config 2024-03-29T11:18:30.156 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.179+0000 7fa41579d640 10 monclient: build_initial_monmap 2024-03-29T11:18:30.156 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.179+0000 7fa41579d640 10 monclient: monmap: 2024-03-29T11:18:30.156 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:18:30.156 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:18:30.156 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:18:30.180291+0000 2024-03-29T11:18:30.157 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:18:30.180291+0000 2024-03-29T11:18:30.157 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:30.157 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:30.157 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:18:30.157 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:18:30.157 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:18:30.157 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:30.157 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.180+0000 7fa41579d640 1 Processor -- start 2024-03-29T11:18:30.158 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.180+0000 7fa41579d640 1 -- start start 2024-03-29T11:18:30.158 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.180+0000 7fa41579d640 10 monclient: init 2024-03-29T11:18:30.159 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.181+0000 7fa41579d640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:30.159 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.181+0000 7fa41579d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:18:30.159 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.181+0000 7fa41579d640 10 monclient(hunting): picked mon.noname-a con 0x7fa41006f050 addr v1:172.21.15.47:6789/0 2024-03-29T11:18:30.159 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.181+0000 7fa41579d640 10 monclient(hunting): picked mon.noname-b con 0x7fa410070240 addr v1:172.21.15.47:6790/0 2024-03-29T11:18:30.159 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.181+0000 7fa41579d640 10 monclient(hunting): picked mon.noname-c con 0x7fa41006f430 addr v1:172.21.15.121:6789/0 2024-03-29T11:18:30.159 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.181+0000 7fa41579d640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa41013f400 con 0x7fa41006f050 2024-03-29T11:18:30.159 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.181+0000 7fa41579d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4101429c0 con 0x7fa41006f430 2024-03-29T11:18:30.159 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.181+0000 7fa41579d640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa41006ff80 con 0x7fa410070240 2024-03-29T11:18:30.159 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.181+0000 7fa41579d640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:30.159 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.181+0000 7fa41579d640 10 monclient(hunting): authenticate will time out at 1153.369646s 2024-03-29T11:18:30.159 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.181+0000 7fa40effd640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fa41006f050 0x7fa4101422b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:48102/0 (socket says 172.21.15.47:48102) 2024-03-29T11:18:30.159 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.181+0000 7fa40effd640 1 -- 172.21.15.47:0/2842270899 learned_addr learned my addr 172.21.15.47:0/2842270899 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:18:30.159 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.182+0000 7fa40dffb640 1 -- 172.21.15.47:0/2842270899 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2893766657 0 0) 0x7fa41013f400 con 0x7fa41006f050 2024-03-29T11:18:30.160 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.182+0000 7fa40dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:18:30.160 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.182+0000 7fa40dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:30.160 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.182+0000 7fa40dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:30.160 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.182+0000 7fa40dffb640 10 monclient(hunting): my global_id is 4407 2024-03-29T11:18:30.160 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.182+0000 7fa40dffb640 1 -- 172.21.15.47:0/2842270899 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3f8003520 con 0x7fa41006f050 2024-03-29T11:18:30.160 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.182+0000 7fa40dffb640 1 -- 172.21.15.47:0/2842270899 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3203787060 0 0) 0x7fa3f8003520 con 0x7fa41006f050 2024-03-29T11:18:30.160 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.182+0000 7fa40dffb640 1 -- 172.21.15.47:0/2842270899 >> v1:172.21.15.47:6790/0 conn(0x7fa410070240 legacy=0x7fa4100ae860 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:18:30.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.184+0000 7fa40dffb640 1 -- 172.21.15.47:0/2842270899 >> v1:172.21.15.121:6789/0 conn(0x7fa41006f430 legacy=0x7fa41014a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:30.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.184+0000 7fa40dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:18:30.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.184+0000 7fa40dffb640 1 monclient: found mon.noname-a 2024-03-29T11:18:30.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.184+0000 7fa40dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:30.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.184+0000 7fa40dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-29T11:18:30.162 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.184+0000 7fa40dffb640 1 -- 172.21.15.47:0/2842270899 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4100aef70 con 0x7fa41006f050 2024-03-29T11:18:30.162 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.184+0000 7fa40dffb640 10 monclient: _finish_auth 0 2024-03-29T11:18:30.162 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.184+0000 7fa40dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:18:30.162 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.184+0000 7fa40dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:30.162 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.184+0000 7fa40dffb640 1 -- 172.21.15.47:0/2842270899 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fa400001690 con 0x7fa41006f050 2024-03-29T11:18:30.162 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.184+0000 7fa40dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:30.162 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.184+0000 7fa40dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:18:30.162 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.184+0000 7fa40dffb640 10 monclient: dump: 2024-03-29T11:18:30.162 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:18:30.162 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:18:30.162 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:30.162 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:30.162 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:30.162 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:30.163 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:18:30.163 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:18:30.163 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:18:30.163 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:30.163 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.184+0000 7fa41579d640 5 monclient: authenticate success, global_id 4407 2024-03-29T11:18:30.163 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.184+0000 7fa41579d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:30.164 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.185+0000 7fa40dffb640 1 -- 172.21.15.47:0/2842270899 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa400001860 con 0x7fa41006f050 2024-03-29T11:18:30.164 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.185+0000 7fa40dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:30.165 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.185+0000 7fa40dffb640 1 -- 172.21.15.47:0/2842270899 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fa400001cb0 con 0x7fa41006f050 2024-03-29T11:18:30.165 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.185+0000 7fa41579d640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:30.165 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.185+0000 7fa41579d640 4 set_mon_vals no callback set 2024-03-29T11:18:30.165 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.185+0000 7fa40dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:30.165 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.185+0000 7fa40dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:18:30.165 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.185+0000 7fa40dffb640 10 monclient: dump: 2024-03-29T11:18:30.165 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:18:30.165 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:18:30.165 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:30.165 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:30.165 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:30.165 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:30.165 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:18:30.165 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:18:30.166 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:18:30.166 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:30.166 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.186+0000 7fa41579d640 10 monclient: shutdown 2024-03-29T11:18:30.166 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.186+0000 7fa41579d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:30.166 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.186+0000 7fa41579d640 1 -- 172.21.15.47:0/2842270899 >> v1:172.21.15.47:6789/0 conn(0x7fa41006f050 legacy=0x7fa4101422b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:30.166 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.186+0000 7fa41579d640 1 -- 172.21.15.47:0/2842270899 shutdown_connections 2024-03-29T11:18:30.166 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.186+0000 7fa41579d640 1 -- 172.21.15.47:0/2842270899 >> 172.21.15.47:0/2842270899 conn(0x7fa4100b4930 msgr2=0x7fa4100b4d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:30.166 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.186+0000 7fa41579d640 1 -- 172.21.15.47:0/2842270899 shutdown_connections 2024-03-29T11:18:30.166 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.186+0000 7fa41579d640 1 -- 172.21.15.47:0/2842270899 wait complete. 2024-03-29T11:18:30.166 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.187+0000 7fa41579d640 10 monclient: build_initial_monmap 2024-03-29T11:18:30.166 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.187+0000 7fa41579d640 10 monclient: monmap: 2024-03-29T11:18:30.166 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:18:30.166 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:30.166 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:18:30.167 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:18:30.167 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:30.167 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:30.167 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:18:30.167 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:18:30.167 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:18:30.167 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:30.167 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.187+0000 7fa41579d640 1 librados: starting msgr at 2024-03-29T11:18:30.167 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.187+0000 7fa41579d640 1 librados: starting objecter 2024-03-29T11:18:30.167 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.187+0000 7fa41579d640 1 Processor -- start 2024-03-29T11:18:30.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.188+0000 7fa41579d640 1 -- start start 2024-03-29T11:18:30.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.188+0000 7fa41579d640 1 librados: setting wanted keys 2024-03-29T11:18:30.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.188+0000 7fa41579d640 1 librados: calling monclient init 2024-03-29T11:18:30.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.188+0000 7fa41579d640 10 monclient: init 2024-03-29T11:18:30.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.188+0000 7fa41579d640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:30.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.188+0000 7fa41579d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:18:30.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.189+0000 7fa41579d640 10 monclient(hunting): picked mon.noname-c con 0x7fa41006f050 addr v1:172.21.15.47:6790/0 2024-03-29T11:18:30.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.189+0000 7fa41579d640 10 monclient(hunting): picked mon.noname-a con 0x7fa41006f430 addr v1:172.21.15.47:6789/0 2024-03-29T11:18:30.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.189+0000 7fa41579d640 10 monclient(hunting): picked mon.noname-b con 0x7fa410070240 addr v1:172.21.15.121:6789/0 2024-03-29T11:18:30.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.189+0000 7fa41579d640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4100abd50 con 0x7fa41006f430 2024-03-29T11:18:30.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.189+0000 7fa41579d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa41013b3a0 con 0x7fa410070240 2024-03-29T11:18:30.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.189+0000 7fa41579d640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa41013b530 con 0x7fa41006f050 2024-03-29T11:18:30.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.189+0000 7fa41579d640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:30.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.189+0000 7fa41579d640 10 monclient(hunting): authenticate will time out at 1153.377698s 2024-03-29T11:18:30.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.189+0000 7fa40e7fc640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fa41006f430 0x7fa4100ab640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:50992/0 (socket says 172.21.15.47:50992) 2024-03-29T11:18:30.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.189+0000 7fa40e7fc640 1 -- 172.21.15.47:0/4065725791 learned_addr learned my addr 172.21.15.47:0/4065725791 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:18:30.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.190+0000 7fa3f77fe640 10 client.?.objecter ms_handle_connect 0x7fa41006f430 2024-03-29T11:18:30.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.190+0000 7fa3f77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:30.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.190+0000 7fa3f77fe640 1 -- 172.21.15.47:0/4065725791 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2454508854 0 0) 0x7fa4100abd50 con 0x7fa41006f430 2024-03-29T11:18:30.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.190+0000 7fa3f77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:18:30.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.190+0000 7fa3f77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:30.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.190+0000 7fa3f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:30.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.190+0000 7fa3f77fe640 10 monclient(hunting): my global_id is 4410 2024-03-29T11:18:30.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.190+0000 7fa3f77fe640 1 -- 172.21.15.47:0/4065725791 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3fc004eb0 con 0x7fa41006f430 2024-03-29T11:18:30.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.190+0000 7fa3f77fe640 10 client.?.objecter ms_handle_connect 0x7fa410070240 2024-03-29T11:18:30.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.190+0000 7fa3f77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:30.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.190+0000 7fa3f77fe640 1 -- 172.21.15.47:0/4065725791 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 163629589 0 0) 0x7fa41013b3a0 con 0x7fa410070240 2024-03-29T11:18:30.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.190+0000 7fa3f77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:18:30.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.190+0000 7fa3f77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:30.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.190+0000 7fa3f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:30.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.190+0000 7fa3f77fe640 10 monclient(hunting): my global_id is 4417 2024-03-29T11:18:30.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.190+0000 7fa3f77fe640 1 -- 172.21.15.47:0/4065725791 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4100abd50 con 0x7fa410070240 2024-03-29T11:18:30.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.190+0000 7fa3f77fe640 1 -- 172.21.15.47:0/4065725791 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3654897852 0 0) 0x7fa3fc004eb0 con 0x7fa41006f430 2024-03-29T11:18:30.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.190+0000 7fa3f77fe640 1 -- 172.21.15.47:0/4065725791 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa41013b3a0 con 0x7fa41006f430 2024-03-29T11:18:30.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.190+0000 7fa3f77fe640 1 -- 172.21.15.47:0/4065725791 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fa4040017d0 con 0x7fa41006f430 2024-03-29T11:18:30.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.190+0000 7fa3f77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:30.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.190+0000 7fa3f77fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:18:30.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.190+0000 7fa3f77fe640 10 monclient(hunting): dump: 2024-03-29T11:18:30.174 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:18:30.175 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:18:30.175 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:30.175 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:30.175 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:30.175 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:30.175 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:18:30.175 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:18:30.175 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:18:30.175 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:30.175 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.190+0000 7fa3f77fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:30.175 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.190+0000 7fa3f77fe640 1 -- 172.21.15.47:0/4065725791 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 975924837 0 0) 0x7fa4100abd50 con 0x7fa410070240 2024-03-29T11:18:30.175 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.190+0000 7fa3f77fe640 1 -- 172.21.15.47:0/4065725791 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa3fc004eb0 con 0x7fa410070240 2024-03-29T11:18:30.175 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.190+0000 7fa3f77fe640 1 -- 172.21.15.47:0/4065725791 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fa408001bb0 con 0x7fa410070240 2024-03-29T11:18:30.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.190+0000 7fa3f77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:30.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.190+0000 7fa3f77fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:18:30.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.190+0000 7fa3f77fe640 10 monclient(hunting): dump: 2024-03-29T11:18:30.176 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:18:30.176 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:18:30.176 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:30.176 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:30.176 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:30.177 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:30.177 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:18:30.177 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:18:30.177 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:18:30.177 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:30.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.190+0000 7fa3f77fe640 1 -- 172.21.15.47:0/4065725791 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1182091351 0 0) 0x7fa41013b3a0 con 0x7fa41006f430 2024-03-29T11:18:30.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.190+0000 7fa3f77fe640 1 -- 172.21.15.47:0/4065725791 >> v1:172.21.15.47:6790/0 conn(0x7fa41006f050 legacy=0x7fa4100abf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:30.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.190+0000 7fa3f77fe640 1 -- 172.21.15.47:0/4065725791 >> v1:172.21.15.121:6789/0 conn(0x7fa410070240 legacy=0x7fa4101e5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:30.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.190+0000 7fa3f77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:18:30.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.190+0000 7fa3f77fe640 1 monclient: found mon.a 2024-03-29T11:18:30.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.190+0000 7fa3f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:30.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.190+0000 7fa3f77fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:18:30.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.190+0000 7fa3f77fe640 1 -- 172.21.15.47:0/4065725791 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa410141bd0 con 0x7fa41006f430 2024-03-29T11:18:30.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.190+0000 7fa3f77fe640 10 monclient: _finish_auth 0 2024-03-29T11:18:30.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.190+0000 7fa3f77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:18:30.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.190+0000 7fa3f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:30.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.190+0000 7fa41579d640 5 monclient: authenticate success, global_id 4410 2024-03-29T11:18:30.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.190+0000 7fa41579d640 10 monclient: _renew_subs 2024-03-29T11:18:30.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.190+0000 7fa41579d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:18:30.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.190+0000 7fa41579d640 1 -- 172.21.15.47:0/4065725791 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa410141ea0 con 0x7fa41006f430 2024-03-29T11:18:30.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.190+0000 7fa41579d640 10 client.4410.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:30.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.190+0000 7fa41579d640 10 monclient: _renew_subs 2024-03-29T11:18:30.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.190+0000 7fa41579d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:18:30.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.190+0000 7fa41579d640 1 -- 172.21.15.47:0/4065725791 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4101de830 con 0x7fa41006f430 2024-03-29T11:18:30.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.190+0000 7fa41579d640 1 librados: init done 2024-03-29T11:18:30.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.191+0000 7fa3f77fe640 1 -- 172.21.15.47:0/4065725791 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa404001ce0 con 0x7fa41006f430 2024-03-29T11:18:30.180 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.191+0000 7fa3f77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:30.180 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.191+0000 7fa3f77fe640 1 -- 172.21.15.47:0/4065725791 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fa404004a00 con 0x7fa41006f430 2024-03-29T11:18:30.180 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.191+0000 7fa3f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:30.180 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.191+0000 7fa3f77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:18:30.180 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.191+0000 7fa3f77fe640 10 monclient: dump: 2024-03-29T11:18:30.180 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:18:30.180 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:18:30.181 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:30.181 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:30.181 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:30.181 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:30.181 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:18:30.181 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:18:30.181 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:18:30.181 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:30.181 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.191+0000 7fa40d7fa640 4 set_mon_vals no callback set 2024-03-29T11:18:30.181 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.192+0000 7fa41579d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:30.181 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.193+0000 7fa41579d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:30.182 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.193+0000 7fa41579d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:18:30.182 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.193+0000 7fa41579d640 1 -- 172.21.15.47:0/4065725791 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3f0005080 con 0x7fa41006f430 2024-03-29T11:18:30.182 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.193+0000 7fa3f77fe640 1 -- 172.21.15.47:0/4065725791 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7fa40401a9d0 con 0x7fa41006f430 2024-03-29T11:18:30.182 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.194+0000 7fa3f77fe640 1 -- 172.21.15.47:0/4065725791 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7fa4040849f0 con 0x7fa41006f430 2024-03-29T11:18:30.182 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.194+0000 7fa3f77fe640 10 client.4410.objecter ms_dispatch 0x7fa410000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:18:30.182 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.194+0000 7fa3f77fe640 3 client.4410.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:18:30.182 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.194+0000 7fa3f77fe640 3 client.4410.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:18:30.182 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.194+0000 7fa3f77fe640 20 client.4410.objecter dump_active .. 0 homeless 2024-03-29T11:18:30.182 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.194+0000 7fa40effd640 10 monclient: get_auth_request con 0x7fa3fc06dce0 auth_method 0 2024-03-29T11:18:30.182 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.194+0000 7fa3f77fe640 10 client.4410.objecter ms_handle_connect 0x7fa3fc06dce0 2024-03-29T11:18:30.183 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.201+0000 7fa3f77fe640 1 -- 172.21.15.47:0/4065725791 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa4040502d0 con 0x7fa41006f430 2024-03-29T11:18:30.183 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.201+0000 7fa3f77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:30.184 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.201+0000 7fa3f77fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:30.510 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.531+0000 7fa41579d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:30.510 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.531+0000 7fa41579d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:30.510 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.531+0000 7fa41579d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:18:30.510 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.531+0000 7fa41579d640 1 -- 172.21.15.47:0/4065725791 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa3f0005330 con 0x7fa41006f430 2024-03-29T11:18:30.510 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.532+0000 7fa3f77fe640 1 -- 172.21.15.47:0/4065725791 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7fa404052200 con 0x7fa41006f430 2024-03-29T11:18:30.510 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.533+0000 7fa3f77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:30.510 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.533+0000 7fa3f77fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:30.510 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:18:30.511 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:18:30.512 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.535+0000 7fa41579d640 10 librados: watch_flush enter 2024-03-29T11:18:30.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.535+0000 7fa41579d640 10 librados: watch_flush exit 2024-03-29T11:18:30.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.535+0000 7fa41579d640 20 client.4410.objecter shutdown clearing up homeless session... 2024-03-29T11:18:30.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.536+0000 7fa41579d640 10 client.4410.objecter successfully canceled tick 2024-03-29T11:18:30.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.536+0000 7fa41579d640 1 -- 172.21.15.47:0/4065725791 >> v1:172.21.15.121:6816/1964117370 conn(0x7fa3fc06dce0 legacy=0x7fa3fc0701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:30.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.536+0000 7fa41579d640 10 monclient: shutdown 2024-03-29T11:18:30.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.536+0000 7fa41579d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:30.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.536+0000 7fa41579d640 1 -- 172.21.15.47:0/4065725791 >> v1:172.21.15.47:6789/0 conn(0x7fa41006f430 legacy=0x7fa4100ab640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:30.514 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.536+0000 7fa41579d640 1 -- 172.21.15.47:0/4065725791 shutdown_connections 2024-03-29T11:18:30.514 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.536+0000 7fa41579d640 1 -- 172.21.15.47:0/4065725791 >> 172.21.15.47:0/4065725791 conn(0x7fa4100b4930 msgr2=0x7fa4100adfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:30.514 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.537+0000 7fa41579d640 1 -- 172.21.15.47:0/4065725791 shutdown_connections 2024-03-29T11:18:30.514 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.537+0000 7fa41579d640 1 -- 172.21.15.47:0/4065725791 wait complete. 2024-03-29T11:18:30.514 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:30.537+0000 7fa41579d640 1 librados: shutdown 2024-03-29T11:18:30.529 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:18:37.335 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:18:37.482 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.504+0000 7f6e61230640 10 monclient: get_monmap_and_config 2024-03-29T11:18:37.483 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.504+0000 7f6e61230640 10 monclient: build_initial_monmap 2024-03-29T11:18:37.483 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.504+0000 7f6e61230640 10 monclient: monmap: 2024-03-29T11:18:37.483 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:18:37.483 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:18:37.483 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:18:37.505365+0000 2024-03-29T11:18:37.483 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:18:37.505365+0000 2024-03-29T11:18:37.483 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:37.483 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:37.483 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:18:37.484 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:18:37.484 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:18:37.484 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:37.484 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.505+0000 7f6e61230640 1 Processor -- start 2024-03-29T11:18:37.484 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.505+0000 7f6e61230640 1 -- start start 2024-03-29T11:18:37.484 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.505+0000 7f6e61230640 10 monclient: init 2024-03-29T11:18:37.484 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.506+0000 7f6e61230640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:37.485 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.506+0000 7f6e61230640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:18:37.485 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.506+0000 7f6e61230640 10 monclient(hunting): picked mon.noname-a con 0x7f6e54034850 addr v1:172.21.15.47:6789/0 2024-03-29T11:18:37.485 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.506+0000 7f6e61230640 10 monclient(hunting): picked mon.noname-c con 0x7f6e5402a640 addr v1:172.21.15.121:6789/0 2024-03-29T11:18:37.485 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.506+0000 7f6e61230640 10 monclient(hunting): picked mon.noname-b con 0x7f6e540327d0 addr v1:172.21.15.47:6790/0 2024-03-29T11:18:37.485 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.506+0000 7f6e61230640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e5402c820 con 0x7f6e54034850 2024-03-29T11:18:37.485 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.506+0000 7f6e61230640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e5402a120 con 0x7f6e5402a640 2024-03-29T11:18:37.485 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.506+0000 7f6e61230640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e540c75b0 con 0x7f6e540327d0 2024-03-29T11:18:37.485 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.506+0000 7f6e61230640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:37.485 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.506+0000 7f6e61230640 10 monclient(hunting): authenticate will time out at 1160.694733s 2024-03-29T11:18:37.485 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.507+0000 7f6e5b59e640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f6e540327d0 0x7f6e540cf240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:45150/0 (socket says 172.21.15.47:45150) 2024-03-29T11:18:37.486 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.507+0000 7f6e5b59e640 1 -- 172.21.15.47:0/1231944523 learned_addr learned my addr 172.21.15.47:0/1231944523 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:18:37.486 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.508+0000 7f6e59d9b640 1 -- 172.21.15.47:0/1231944523 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 657712446 0 0) 0x7f6e540c75b0 con 0x7f6e540327d0 2024-03-29T11:18:37.486 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.508+0000 7f6e59d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:18:37.486 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.508+0000 7f6e59d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:37.486 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.508+0000 7f6e59d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:37.486 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.508+0000 7f6e59d9b640 10 monclient(hunting): my global_id is 4454 2024-03-29T11:18:37.487 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.508+0000 7f6e59d9b640 1 -- 172.21.15.47:0/1231944523 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e44003520 con 0x7f6e540327d0 2024-03-29T11:18:37.491 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.510+0000 7f6e59d9b640 1 -- 172.21.15.47:0/1231944523 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1228338169 0 0) 0x7f6e44003520 con 0x7f6e540327d0 2024-03-29T11:18:37.491 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.510+0000 7f6e59d9b640 1 -- 172.21.15.47:0/1231944523 >> v1:172.21.15.121:6789/0 conn(0x7f6e5402a640 legacy=0x7f6e540320c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:18:37.491 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.510+0000 7f6e59d9b640 1 -- 172.21.15.47:0/1231944523 >> v1:172.21.15.47:6789/0 conn(0x7f6e54034850 legacy=0x7f6e540c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:37.491 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.510+0000 7f6e59d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:18:37.491 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.510+0000 7f6e59d9b640 1 monclient: found mon.noname-b 2024-03-29T11:18:37.492 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.510+0000 7f6e59d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:37.492 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.510+0000 7f6e59d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-29T11:18:37.492 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.510+0000 7f6e59d9b640 1 -- 172.21.15.47:0/1231944523 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e54033110 con 0x7f6e540327d0 2024-03-29T11:18:37.492 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.510+0000 7f6e59d9b640 10 monclient: _finish_auth 0 2024-03-29T11:18:37.492 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.510+0000 7f6e59d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:18:37.492 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.510+0000 7f6e59d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:37.492 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.510+0000 7f6e59d9b640 1 -- 172.21.15.47:0/1231944523 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f6e4c001690 con 0x7f6e540327d0 2024-03-29T11:18:37.492 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.510+0000 7f6e59d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:37.492 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.510+0000 7f6e59d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:18:37.492 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.510+0000 7f6e59d9b640 10 monclient: dump: 2024-03-29T11:18:37.493 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:18:37.493 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:18:37.493 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:37.493 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:37.493 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:37.493 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:37.493 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:18:37.493 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:18:37.493 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:18:37.493 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:37.494 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.510+0000 7f6e61230640 5 monclient: authenticate success, global_id 4454 2024-03-29T11:18:37.494 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.510+0000 7f6e61230640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:37.494 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.510+0000 7f6e59d9b640 1 -- 172.21.15.47:0/1231944523 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e4c001860 con 0x7f6e540327d0 2024-03-29T11:18:37.494 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.510+0000 7f6e59d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:37.494 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.510+0000 7f6e59d9b640 1 -- 172.21.15.47:0/1231944523 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f6e4c001cb0 con 0x7f6e540327d0 2024-03-29T11:18:37.494 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.510+0000 7f6e59d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:37.494 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.510+0000 7f6e59d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:37.494 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.510+0000 7f6e59d9b640 10 monclient: dump: 2024-03-29T11:18:37.494 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:18:37.494 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:18:37.495 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:37.495 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:37.495 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:37.495 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:37.495 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:18:37.495 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:18:37.495 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:18:37.495 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:37.495 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.510+0000 7f6e61230640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:37.495 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.510+0000 7f6e61230640 4 set_mon_vals no callback set 2024-03-29T11:18:37.496 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.511+0000 7f6e61230640 10 monclient: shutdown 2024-03-29T11:18:37.496 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.511+0000 7f6e61230640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:37.496 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.511+0000 7f6e61230640 1 -- 172.21.15.47:0/1231944523 >> v1:172.21.15.47:6790/0 conn(0x7f6e540327d0 legacy=0x7f6e540cf240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:37.496 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.511+0000 7f6e61230640 1 -- 172.21.15.47:0/1231944523 shutdown_connections 2024-03-29T11:18:37.496 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.511+0000 7f6e61230640 1 -- 172.21.15.47:0/1231944523 >> 172.21.15.47:0/1231944523 conn(0x7f6e540058f0 msgr2=0x7f6e54005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:37.496 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.511+0000 7f6e61230640 1 -- 172.21.15.47:0/1231944523 shutdown_connections 2024-03-29T11:18:37.496 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.511+0000 7f6e61230640 1 -- 172.21.15.47:0/1231944523 wait complete. 2024-03-29T11:18:37.497 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.514+0000 7f6e61230640 10 monclient: build_initial_monmap 2024-03-29T11:18:37.497 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.514+0000 7f6e61230640 10 monclient: monmap: 2024-03-29T11:18:37.497 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:18:37.497 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:37.497 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:18:37.497 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:18:37.498 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:37.498 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:37.498 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:18:37.498 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:18:37.498 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:18:37.498 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:37.498 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.514+0000 7f6e61230640 1 librados: starting msgr at 2024-03-29T11:18:37.498 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.514+0000 7f6e61230640 1 librados: starting objecter 2024-03-29T11:18:37.498 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.515+0000 7f6e61230640 1 Processor -- start 2024-03-29T11:18:37.499 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.515+0000 7f6e61230640 1 -- start start 2024-03-29T11:18:37.499 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.515+0000 7f6e61230640 1 librados: setting wanted keys 2024-03-29T11:18:37.499 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.515+0000 7f6e61230640 1 librados: calling monclient init 2024-03-29T11:18:37.499 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.515+0000 7f6e61230640 10 monclient: init 2024-03-29T11:18:37.499 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.515+0000 7f6e61230640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:37.499 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.515+0000 7f6e61230640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:18:37.499 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.515+0000 7f6e61230640 10 monclient(hunting): picked mon.noname-b con 0x7f6e5402a640 addr v1:172.21.15.121:6789/0 2024-03-29T11:18:37.499 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.515+0000 7f6e61230640 10 monclient(hunting): picked mon.noname-c con 0x7f6e540327d0 addr v1:172.21.15.47:6790/0 2024-03-29T11:18:37.499 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.515+0000 7f6e61230640 10 monclient(hunting): picked mon.noname-a con 0x7f6e54034850 addr v1:172.21.15.47:6789/0 2024-03-29T11:18:37.499 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.515+0000 7f6e61230640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e540d2460 con 0x7f6e54034850 2024-03-29T11:18:37.500 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.515+0000 7f6e61230640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e540d2180 con 0x7f6e5402a640 2024-03-29T11:18:37.500 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.515+0000 7f6e61230640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e540d4370 con 0x7f6e540327d0 2024-03-29T11:18:37.500 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.515+0000 7f6e61230640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:37.500 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.515+0000 7f6e61230640 10 monclient(hunting): authenticate will time out at 1160.703861s 2024-03-29T11:18:37.500 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.516+0000 7f6e5a59c640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f6e540327d0 0x7f6e540bc2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:49842/0 (socket says 172.21.15.47:49842) 2024-03-29T11:18:37.500 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.516+0000 7f6e5a59c640 1 -- 172.21.15.47:0/3638244439 learned_addr learned my addr 172.21.15.47:0/3638244439 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:18:37.500 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.518+0000 7f6e437fe640 10 client.?.objecter ms_handle_connect 0x7f6e540327d0 2024-03-29T11:18:37.505 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.518+0000 7f6e437fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:37.506 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.518+0000 7f6e437fe640 1 -- 172.21.15.47:0/3638244439 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3005792464 0 0) 0x7f6e540d4370 con 0x7f6e540327d0 2024-03-29T11:18:37.506 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.518+0000 7f6e437fe640 10 monclient(hunting): _negotiate 2024-03-29T11:18:37.506 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.518+0000 7f6e437fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:37.506 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.518+0000 7f6e437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:37.506 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.518+0000 7f6e437fe640 10 monclient(hunting): my global_id is 4457 2024-03-29T11:18:37.506 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.518+0000 7f6e437fe640 1 -- 172.21.15.47:0/3638244439 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e48004eb0 con 0x7f6e540327d0 2024-03-29T11:18:37.506 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.518+0000 7f6e437fe640 1 -- 172.21.15.47:0/3638244439 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1066412245 0 0) 0x7f6e48004eb0 con 0x7f6e540327d0 2024-03-29T11:18:37.507 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.518+0000 7f6e437fe640 1 -- 172.21.15.47:0/3638244439 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e540d4370 con 0x7f6e540327d0 2024-03-29T11:18:37.507 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.518+0000 7f6e437fe640 1 -- 172.21.15.47:0/3638244439 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f6e5c04fdd0 con 0x7f6e540327d0 2024-03-29T11:18:37.507 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.518+0000 7f6e437fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:37.507 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.518+0000 7f6e437fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:18:37.507 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.518+0000 7f6e437fe640 10 monclient(hunting): dump: 2024-03-29T11:18:37.507 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:18:37.507 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:18:37.507 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:37.507 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:37.507 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:37.508 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:37.508 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:18:37.508 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:18:37.508 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:18:37.508 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:37.508 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.518+0000 7f6e437fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:37.508 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.518+0000 7f6e437fe640 1 -- 172.21.15.47:0/3638244439 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3942062595 0 0) 0x7f6e540d4370 con 0x7f6e540327d0 2024-03-29T11:18:37.508 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.519+0000 7f6e437fe640 1 -- 172.21.15.47:0/3638244439 >> v1:172.21.15.121:6789/0 conn(0x7f6e5402a640 legacy=0x7f6e540d2cd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:18:37.509 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.522+0000 7f6e437fe640 1 -- 172.21.15.47:0/3638244439 >> v1:172.21.15.47:6789/0 conn(0x7f6e54034850 legacy=0x7f6e540bca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:37.509 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.522+0000 7f6e437fe640 10 monclient: _finish_hunting 0 2024-03-29T11:18:37.509 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.522+0000 7f6e437fe640 1 monclient: found mon.c 2024-03-29T11:18:37.509 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.522+0000 7f6e437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:37.509 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.522+0000 7f6e437fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:18:37.509 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.522+0000 7f6e437fe640 1 -- 172.21.15.47:0/3638244439 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e540d4650 con 0x7f6e540327d0 2024-03-29T11:18:37.509 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.522+0000 7f6e437fe640 10 monclient: _finish_auth 0 2024-03-29T11:18:37.509 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.522+0000 7f6e437fe640 10 monclient: _check_auth_tickets 2024-03-29T11:18:37.510 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.522+0000 7f6e437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:37.510 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.522+0000 7f6e61230640 5 monclient: authenticate success, global_id 4457 2024-03-29T11:18:37.510 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.522+0000 7f6e61230640 10 monclient: _renew_subs 2024-03-29T11:18:37.510 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.522+0000 7f6e61230640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:18:37.510 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.522+0000 7f6e61230640 1 -- 172.21.15.47:0/3638244439 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e540bd110 con 0x7f6e540327d0 2024-03-29T11:18:37.510 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.522+0000 7f6e61230640 10 client.4457.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:37.510 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.522+0000 7f6e61230640 10 monclient: _renew_subs 2024-03-29T11:18:37.510 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.522+0000 7f6e61230640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:18:37.510 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.522+0000 7f6e61230640 1 -- 172.21.15.47:0/3638244439 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e540d2920 con 0x7f6e540327d0 2024-03-29T11:18:37.511 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.522+0000 7f6e61230640 1 librados: init done 2024-03-29T11:18:37.511 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.523+0000 7f6e61230640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:37.511 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.523+0000 7f6e437fe640 1 -- 172.21.15.47:0/3638244439 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e5c04f400 con 0x7f6e540327d0 2024-03-29T11:18:37.511 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.523+0000 7f6e61230640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:37.511 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.523+0000 7f6e61230640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:18:37.511 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.523+0000 7f6e61230640 1 -- 172.21.15.47:0/3638244439 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e54004cd0 con 0x7f6e540327d0 2024-03-29T11:18:37.511 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.523+0000 7f6e437fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:37.511 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.523+0000 7f6e437fe640 1 -- 172.21.15.47:0/3638244439 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f6e5c075f10 con 0x7f6e540327d0 2024-03-29T11:18:37.511 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.523+0000 7f6e437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:37.511 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.523+0000 7f6e437fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:37.511 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.523+0000 7f6e437fe640 10 monclient: dump: 2024-03-29T11:18:37.512 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:18:37.512 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:18:37.512 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:37.512 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:37.512 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:37.512 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:37.512 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:18:37.512 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:18:37.512 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:18:37.512 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:37.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.523+0000 7f6e5959a640 4 set_mon_vals no callback set 2024-03-29T11:18:37.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.525+0000 7f6e437fe640 1 -- 172.21.15.47:0/3638244439 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7f6e5c067bf0 con 0x7f6e540327d0 2024-03-29T11:18:37.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.526+0000 7f6e437fe640 1 -- 172.21.15.47:0/3638244439 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7f6e5c110680 con 0x7f6e540327d0 2024-03-29T11:18:37.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.526+0000 7f6e437fe640 10 client.4457.objecter ms_dispatch 0x7f6e5c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:18:37.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.526+0000 7f6e437fe640 3 client.4457.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:18:37.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.526+0000 7f6e437fe640 3 client.4457.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:18:37.513 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.526+0000 7f6e437fe640 20 client.4457.objecter dump_active .. 0 homeless 2024-03-29T11:18:37.514 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.529+0000 7f6e5ad9d640 10 monclient: get_auth_request con 0x7f6e4806dba0 auth_method 0 2024-03-29T11:18:37.515 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.532+0000 7f6e437fe640 1 -- 172.21.15.47:0/3638244439 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6e5c0dbf60 con 0x7f6e540327d0 2024-03-29T11:18:37.515 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.532+0000 7f6e437fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:37.515 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.532+0000 7f6e437fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:37.515 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.535+0000 7f6e437fe640 10 client.4457.objecter ms_handle_connect 0x7f6e4806dba0 2024-03-29T11:18:37.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.876+0000 7f6e61230640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:37.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.876+0000 7f6e61230640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:37.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.876+0000 7f6e61230640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:18:37.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.876+0000 7f6e61230640 1 -- 172.21.15.47:0/3638244439 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e540086f0 con 0x7f6e540327d0 2024-03-29T11:18:37.854 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.877+0000 7f6e437fe640 1 -- 172.21.15.47:0/3638244439 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7f6e5c0dde90 con 0x7f6e540327d0 2024-03-29T11:18:37.854 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.877+0000 7f6e437fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:37.854 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.877+0000 7f6e437fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:37.854 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:18:37.854 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:18:37.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.879+0000 7f6e61230640 10 librados: watch_flush enter 2024-03-29T11:18:37.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.879+0000 7f6e61230640 10 librados: watch_flush exit 2024-03-29T11:18:37.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.879+0000 7f6e61230640 20 client.4457.objecter shutdown clearing up homeless session... 2024-03-29T11:18:37.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.879+0000 7f6e61230640 10 client.4457.objecter successfully canceled tick 2024-03-29T11:18:37.857 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.880+0000 7f6e61230640 1 -- 172.21.15.47:0/3638244439 >> v1:172.21.15.121:6816/1964117370 conn(0x7f6e4806dba0 legacy=0x7f6e48070060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:37.857 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.880+0000 7f6e61230640 10 monclient: shutdown 2024-03-29T11:18:37.857 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.880+0000 7f6e61230640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:37.857 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.880+0000 7f6e61230640 1 -- 172.21.15.47:0/3638244439 >> v1:172.21.15.47:6790/0 conn(0x7f6e540327d0 legacy=0x7f6e540bc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:37.858 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.881+0000 7f6e61230640 1 -- 172.21.15.47:0/3638244439 shutdown_connections 2024-03-29T11:18:37.858 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.881+0000 7f6e61230640 1 -- 172.21.15.47:0/3638244439 >> 172.21.15.47:0/3638244439 conn(0x7f6e540058f0 msgr2=0x7f6e5402aa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:37.858 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.881+0000 7f6e61230640 1 -- 172.21.15.47:0/3638244439 shutdown_connections 2024-03-29T11:18:37.858 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.881+0000 7f6e61230640 1 -- 172.21.15.47:0/3638244439 wait complete. 2024-03-29T11:18:37.858 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:37.881+0000 7f6e61230640 1 librados: shutdown 2024-03-29T11:18:37.877 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:18:44.683 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:18:44.832 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.854+0000 7faa58952640 10 monclient: get_monmap_and_config 2024-03-29T11:18:44.832 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.854+0000 7faa58952640 10 monclient: build_initial_monmap 2024-03-29T11:18:44.832 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.854+0000 7faa58952640 10 monclient: monmap: 2024-03-29T11:18:44.832 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:18:44.833 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:18:44.833 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:18:44.855623+0000 2024-03-29T11:18:44.833 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:18:44.855623+0000 2024-03-29T11:18:44.833 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:44.833 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:44.833 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:18:44.833 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:18:44.833 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:18:44.833 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:44.835 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.855+0000 7faa58952640 1 Processor -- start 2024-03-29T11:18:44.835 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.855+0000 7faa58952640 1 -- start start 2024-03-29T11:18:44.835 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.855+0000 7faa58952640 10 monclient: init 2024-03-29T11:18:44.835 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.856+0000 7faa58952640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:44.835 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.856+0000 7faa58952640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:18:44.835 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.856+0000 7faa58952640 10 monclient(hunting): picked mon.noname-a con 0x7faa540a9ac0 addr v1:172.21.15.47:6789/0 2024-03-29T11:18:44.835 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.856+0000 7faa58952640 10 monclient(hunting): picked mon.noname-c con 0x7faa5413ee90 addr v1:172.21.15.121:6789/0 2024-03-29T11:18:44.836 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.856+0000 7faa58952640 10 monclient(hunting): picked mon.noname-b con 0x7faa540af900 addr v1:172.21.15.47:6790/0 2024-03-29T11:18:44.836 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.857+0000 7faa58952640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa540afd10 con 0x7faa540a9ac0 2024-03-29T11:18:44.836 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.857+0000 7faa58952640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa540af6f0 con 0x7faa5413ee90 2024-03-29T11:18:44.836 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.857+0000 7faa58952640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa540aac80 con 0x7faa540af900 2024-03-29T11:18:44.836 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.857+0000 7faa58952640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:44.836 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.857+0000 7faa58952640 10 monclient(hunting): authenticate will time out at 1168.044977s 2024-03-29T11:18:44.836 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.857+0000 7faa52d9d640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7faa540af900 0x7faa54149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:42086/0 (socket says 172.21.15.47:42086) 2024-03-29T11:18:44.836 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.858+0000 7faa52d9d640 1 -- 172.21.15.47:0/2032552921 learned_addr learned my addr 172.21.15.47:0/2032552921 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:18:44.840 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.860+0000 7faa5159a640 1 -- 172.21.15.47:0/2032552921 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 911788525 0 0) 0x7faa540aac80 con 0x7faa540af900 2024-03-29T11:18:44.840 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.861+0000 7faa5159a640 10 monclient(hunting): _negotiate 2024-03-29T11:18:44.840 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.861+0000 7faa5159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:44.840 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.861+0000 7faa5159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:44.840 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.861+0000 7faa5159a640 10 monclient(hunting): my global_id is 4460 2024-03-29T11:18:44.840 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.861+0000 7faa5159a640 1 -- 172.21.15.47:0/2032552921 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa3c003520 con 0x7faa540af900 2024-03-29T11:18:44.840 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.861+0000 7faa5159a640 1 -- 172.21.15.47:0/2032552921 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3344267035 0 0) 0x7faa540afd10 con 0x7faa540a9ac0 2024-03-29T11:18:44.840 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.861+0000 7faa5159a640 10 monclient(hunting): _negotiate 2024-03-29T11:18:44.841 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.861+0000 7faa5159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:44.841 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.861+0000 7faa5159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:44.841 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.861+0000 7faa5159a640 10 monclient(hunting): my global_id is 4413 2024-03-29T11:18:44.841 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.861+0000 7faa5159a640 1 -- 172.21.15.47:0/2032552921 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa540aac80 con 0x7faa540a9ac0 2024-03-29T11:18:44.841 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.861+0000 7faa5159a640 1 -- 172.21.15.47:0/2032552921 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4029230962 0 0) 0x7faa540aac80 con 0x7faa540a9ac0 2024-03-29T11:18:44.841 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.861+0000 7faa5159a640 1 -- 172.21.15.47:0/2032552921 >> v1:172.21.15.47:6790/0 conn(0x7faa540af900 legacy=0x7faa54149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:44.841 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.861+0000 7faa5159a640 1 -- 172.21.15.47:0/2032552921 >> v1:172.21.15.121:6789/0 conn(0x7faa5413ee90 legacy=0x7faa5413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:18:44.841 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.861+0000 7faa5159a640 10 monclient: _finish_hunting 0 2024-03-29T11:18:44.841 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.861+0000 7faa5159a640 1 monclient: found mon.noname-a 2024-03-29T11:18:44.841 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.861+0000 7faa5159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:44.841 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.862+0000 7faa5159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-29T11:18:44.841 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.862+0000 7faa5159a640 1 -- 172.21.15.47:0/2032552921 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa540aaf60 con 0x7faa540a9ac0 2024-03-29T11:18:44.841 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.862+0000 7faa5159a640 10 monclient: _finish_auth 0 2024-03-29T11:18:44.841 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.862+0000 7faa5159a640 10 monclient: _check_auth_tickets 2024-03-29T11:18:44.842 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.862+0000 7faa5159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:44.842 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.862+0000 7faa5159a640 1 -- 172.21.15.47:0/2032552921 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7faa44001670 con 0x7faa540a9ac0 2024-03-29T11:18:44.842 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.862+0000 7faa58952640 5 monclient: authenticate success, global_id 4413 2024-03-29T11:18:44.842 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.862+0000 7faa58952640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:44.842 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.862+0000 7faa5159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:44.842 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.862+0000 7faa5159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:18:44.842 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.862+0000 7faa5159a640 10 monclient: dump: 2024-03-29T11:18:44.842 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:18:44.842 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:18:44.842 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:44.842 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:44.842 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:44.842 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:44.842 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:18:44.843 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:18:44.843 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:18:44.843 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:44.843 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.862+0000 7faa58952640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:44.843 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.864+0000 7faa5159a640 1 -- 172.21.15.47:0/2032552921 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa440017d0 con 0x7faa540a9ac0 2024-03-29T11:18:44.843 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.864+0000 7faa5159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:44.843 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.864+0000 7faa5159a640 1 -- 172.21.15.47:0/2032552921 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7faa44001c20 con 0x7faa540a9ac0 2024-03-29T11:18:44.843 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.864+0000 7faa5159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:44.843 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.864+0000 7faa5159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:18:44.844 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.864+0000 7faa5159a640 10 monclient: dump: 2024-03-29T11:18:44.844 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:18:44.844 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:18:44.844 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:44.844 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:44.844 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:44.844 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:44.844 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:18:44.844 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:18:44.844 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:18:44.844 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:44.844 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.864+0000 7faa58952640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:44.844 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.864+0000 7faa58952640 4 set_mon_vals no callback set 2024-03-29T11:18:44.845 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.865+0000 7faa58952640 10 monclient: shutdown 2024-03-29T11:18:44.845 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.865+0000 7faa58952640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:44.845 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.865+0000 7faa58952640 1 -- 172.21.15.47:0/2032552921 >> v1:172.21.15.47:6789/0 conn(0x7faa540a9ac0 legacy=0x7faa540adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:44.845 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.866+0000 7faa58952640 1 -- 172.21.15.47:0/2032552921 shutdown_connections 2024-03-29T11:18:44.845 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.866+0000 7faa58952640 1 -- 172.21.15.47:0/2032552921 >> 172.21.15.47:0/2032552921 conn(0x7faa540b35e0 msgr2=0x7faa540b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:44.846 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.867+0000 7faa58952640 1 -- 172.21.15.47:0/2032552921 shutdown_connections 2024-03-29T11:18:44.846 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.868+0000 7faa58952640 1 -- 172.21.15.47:0/2032552921 wait complete. 2024-03-29T11:18:44.846 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.869+0000 7faa58952640 10 monclient: build_initial_monmap 2024-03-29T11:18:44.846 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.869+0000 7faa58952640 10 monclient: monmap: 2024-03-29T11:18:44.846 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:18:44.847 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:44.847 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:18:44.847 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:18:44.847 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:44.847 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:44.847 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:18:44.847 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:18:44.847 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:18:44.847 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:44.847 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.869+0000 7faa58952640 1 librados: starting msgr at 2024-03-29T11:18:44.847 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.869+0000 7faa58952640 1 librados: starting objecter 2024-03-29T11:18:44.847 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.869+0000 7faa58952640 1 Processor -- start 2024-03-29T11:18:44.849 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.870+0000 7faa58952640 1 -- start start 2024-03-29T11:18:44.849 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.870+0000 7faa58952640 1 librados: setting wanted keys 2024-03-29T11:18:44.849 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.870+0000 7faa58952640 1 librados: calling monclient init 2024-03-29T11:18:44.849 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.870+0000 7faa58952640 10 monclient: init 2024-03-29T11:18:44.850 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.870+0000 7faa58952640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:44.850 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.870+0000 7faa58952640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:18:44.850 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.870+0000 7faa58952640 10 monclient(hunting): picked mon.noname-b con 0x7faa540a9ac0 addr v1:172.21.15.121:6789/0 2024-03-29T11:18:44.850 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.870+0000 7faa58952640 10 monclient(hunting): picked mon.noname-c con 0x7faa540af900 addr v1:172.21.15.47:6790/0 2024-03-29T11:18:44.850 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.870+0000 7faa58952640 10 monclient(hunting): picked mon.noname-a con 0x7faa5413ee90 addr v1:172.21.15.47:6789/0 2024-03-29T11:18:44.850 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.870+0000 7faa58952640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa541c8520 con 0x7faa5413ee90 2024-03-29T11:18:44.850 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.870+0000 7faa58952640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa541c7430 con 0x7faa540a9ac0 2024-03-29T11:18:44.850 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.871+0000 7faa58952640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa541c7710 con 0x7faa540af900 2024-03-29T11:18:44.850 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.871+0000 7faa58952640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:44.850 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.871+0000 7faa58952640 10 monclient(hunting): authenticate will time out at 1168.059352s 2024-03-29T11:18:44.850 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.871+0000 7faa51d9b640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7faa540af900 0x7faa541c7e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:43600/0 (socket says 172.21.15.47:43600) 2024-03-29T11:18:44.850 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.871+0000 7faa51d9b640 1 -- 172.21.15.47:0/2517251151 learned_addr learned my addr 172.21.15.47:0/2517251151 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:18:44.852 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.873+0000 7faa42ffd640 10 client.?.objecter ms_handle_connect 0x7faa540af900 2024-03-29T11:18:44.852 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.873+0000 7faa42ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:44.852 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.873+0000 7faa42ffd640 1 -- 172.21.15.47:0/2517251151 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3597077922 0 0) 0x7faa541c7710 con 0x7faa540af900 2024-03-29T11:18:44.852 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.873+0000 7faa42ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:18:44.852 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.873+0000 7faa42ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:44.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.873+0000 7faa42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:44.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.873+0000 7faa42ffd640 10 monclient(hunting): my global_id is 4463 2024-03-29T11:18:44.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.873+0000 7faa42ffd640 1 -- 172.21.15.47:0/2517251151 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa34004eb0 con 0x7faa540af900 2024-03-29T11:18:44.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.874+0000 7faa42ffd640 1 -- 172.21.15.47:0/2517251151 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 822567300 0 0) 0x7faa34004eb0 con 0x7faa540af900 2024-03-29T11:18:44.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.874+0000 7faa42ffd640 1 -- 172.21.15.47:0/2517251151 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa541c7710 con 0x7faa540af900 2024-03-29T11:18:44.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.874+0000 7faa42ffd640 1 -- 172.21.15.47:0/2517251151 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7faa48003fe0 con 0x7faa540af900 2024-03-29T11:18:44.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.875+0000 7faa42ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:44.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.875+0000 7faa42ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:18:44.853 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.875+0000 7faa42ffd640 10 monclient(hunting): dump: 2024-03-29T11:18:44.853 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:18:44.853 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:18:44.853 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:44.853 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:44.853 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:44.854 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:44.854 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:18:44.854 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:18:44.854 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:18:44.854 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:44.854 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.875+0000 7faa42ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:44.854 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.875+0000 7faa42ffd640 1 -- 172.21.15.47:0/2517251151 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 744661980 0 0) 0x7faa541c7710 con 0x7faa540af900 2024-03-29T11:18:44.854 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.875+0000 7faa42ffd640 1 -- 172.21.15.47:0/2517251151 >> v1:172.21.15.121:6789/0 conn(0x7faa540a9ac0 legacy=0x7faa541d8ae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:18:44.855 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.877+0000 7faa42ffd640 1 -- 172.21.15.47:0/2517251151 >> v1:172.21.15.47:6789/0 conn(0x7faa5413ee90 legacy=0x7faa541db5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:44.855 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.877+0000 7faa42ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:18:44.855 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.877+0000 7faa42ffd640 1 monclient: found mon.c 2024-03-29T11:18:44.855 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.878+0000 7faa42ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:44.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.878+0000 7faa42ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:18:44.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.878+0000 7faa42ffd640 1 -- 172.21.15.47:0/2517251151 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa541df3d0 con 0x7faa540af900 2024-03-29T11:18:44.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.878+0000 7faa42ffd640 10 monclient: _finish_auth 0 2024-03-29T11:18:44.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.878+0000 7faa42ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:18:44.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.878+0000 7faa42ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:44.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.878+0000 7faa58952640 5 monclient: authenticate success, global_id 4463 2024-03-29T11:18:44.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.878+0000 7faa58952640 10 monclient: _renew_subs 2024-03-29T11:18:44.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.878+0000 7faa58952640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:18:44.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.878+0000 7faa58952640 1 -- 172.21.15.47:0/2517251151 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa541df8f0 con 0x7faa540af900 2024-03-29T11:18:44.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.878+0000 7faa58952640 10 client.4463.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:44.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.878+0000 7faa58952640 10 monclient: _renew_subs 2024-03-29T11:18:44.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.878+0000 7faa58952640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:18:44.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.878+0000 7faa58952640 1 -- 172.21.15.47:0/2517251151 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faa541c55b0 con 0x7faa540af900 2024-03-29T11:18:44.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.878+0000 7faa58952640 1 librados: init done 2024-03-29T11:18:44.856 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.878+0000 7faa42ffd640 10 client.4463.objecter ms_handle_connect 0x7faa5413ee90 2024-03-29T11:18:44.857 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.878+0000 7faa42ffd640 10 client.4463.objecter resend_mon_ops 2024-03-29T11:18:44.857 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.879+0000 7faa42ffd640 1 -- 172.21.15.47:0/2517251151 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa48000ed0 con 0x7faa540af900 2024-03-29T11:18:44.857 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.879+0000 7faa42ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:44.858 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.879+0000 7faa42ffd640 1 -- 172.21.15.47:0/2517251151 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7faa480012a0 con 0x7faa540af900 2024-03-29T11:18:44.858 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.879+0000 7faa42ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:44.858 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.879+0000 7faa42ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:44.858 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.879+0000 7faa42ffd640 10 monclient: dump: 2024-03-29T11:18:44.858 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:18:44.858 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:18:44.858 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:44.858 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:44.858 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:44.858 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:44.858 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:18:44.858 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:18:44.858 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:18:44.858 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:44.858 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.879+0000 7faa50d99640 4 set_mon_vals no callback set 2024-03-29T11:18:44.859 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.880+0000 7faa58952640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:44.859 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.880+0000 7faa58952640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:44.859 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.880+0000 7faa58952640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:18:44.859 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.880+0000 7faa58952640 1 -- 172.21.15.47:0/2517251151 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa38005080 con 0x7faa540af900 2024-03-29T11:18:44.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.881+0000 7faa42ffd640 1 -- 172.21.15.47:0/2517251151 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7faa48001400 con 0x7faa540af900 2024-03-29T11:18:44.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.882+0000 7faa42ffd640 1 -- 172.21.15.47:0/2517251151 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7faa48084af0 con 0x7faa540af900 2024-03-29T11:18:44.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.882+0000 7faa42ffd640 10 client.4463.objecter ms_dispatch 0x7faa54000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:18:44.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.882+0000 7faa42ffd640 3 client.4463.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:18:44.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.882+0000 7faa42ffd640 3 client.4463.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:18:44.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.882+0000 7faa42ffd640 20 client.4463.objecter dump_active .. 0 homeless 2024-03-29T11:18:44.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.882+0000 7faa5259c640 10 monclient: get_auth_request con 0x7faa3406dba0 auth_method 0 2024-03-29T11:18:44.860 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.883+0000 7faa42ffd640 10 client.4463.objecter ms_handle_connect 0x7faa3406dba0 2024-03-29T11:18:44.866 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.889+0000 7faa42ffd640 1 -- 172.21.15.47:0/2517251151 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faa480503d0 con 0x7faa540af900 2024-03-29T11:18:44.866 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.890+0000 7faa42ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:44.867 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:44.890+0000 7faa42ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:45.206 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:45.229+0000 7faa58952640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:45.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:45.229+0000 7faa58952640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:45.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:45.229+0000 7faa58952640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:18:45.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:45.229+0000 7faa58952640 1 -- 172.21.15.47:0/2517251151 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faa38005330 con 0x7faa540af900 2024-03-29T11:18:45.207 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:45.231+0000 7faa42ffd640 1 -- 172.21.15.47:0/2517251151 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7faa48052300 con 0x7faa540af900 2024-03-29T11:18:45.208 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:45.231+0000 7faa42ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:45.208 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:45.231+0000 7faa42ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:45.208 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:18:45.208 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:18:45.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:45.235+0000 7faa58952640 10 librados: watch_flush enter 2024-03-29T11:18:45.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:45.235+0000 7faa58952640 10 librados: watch_flush exit 2024-03-29T11:18:45.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:45.235+0000 7faa58952640 20 client.4463.objecter shutdown clearing up homeless session... 2024-03-29T11:18:45.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:45.235+0000 7faa58952640 10 client.4463.objecter successfully canceled tick 2024-03-29T11:18:45.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:45.235+0000 7faa58952640 1 -- 172.21.15.47:0/2517251151 >> v1:172.21.15.121:6816/1964117370 conn(0x7faa3406dba0 legacy=0x7faa34070060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:45.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:45.235+0000 7faa58952640 10 monclient: shutdown 2024-03-29T11:18:45.213 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:45.235+0000 7faa58952640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:45.213 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:45.235+0000 7faa58952640 1 -- 172.21.15.47:0/2517251151 >> v1:172.21.15.47:6790/0 conn(0x7faa540af900 legacy=0x7faa541c7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:45.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:45.237+0000 7faa58952640 1 -- 172.21.15.47:0/2517251151 shutdown_connections 2024-03-29T11:18:45.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:45.237+0000 7faa58952640 1 -- 172.21.15.47:0/2517251151 >> 172.21.15.47:0/2517251151 conn(0x7faa540b35e0 msgr2=0x7faa5413a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:45.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:45.239+0000 7faa58952640 1 -- 172.21.15.47:0/2517251151 shutdown_connections 2024-03-29T11:18:45.216 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:45.239+0000 7faa58952640 1 -- 172.21.15.47:0/2517251151 wait complete. 2024-03-29T11:18:45.216 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:45.239+0000 7faa58952640 1 librados: shutdown 2024-03-29T11:18:45.233 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:18:52.040 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:18:52.188 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.210+0000 7f133980f640 10 monclient: get_monmap_and_config 2024-03-29T11:18:52.188 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.210+0000 7f133980f640 10 monclient: build_initial_monmap 2024-03-29T11:18:52.188 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.210+0000 7f133980f640 10 monclient: monmap: 2024-03-29T11:18:52.188 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:18:52.188 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:18:52.188 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:18:52.211427+0000 2024-03-29T11:18:52.188 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:18:52.211427+0000 2024-03-29T11:18:52.188 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:52.188 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:52.189 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:18:52.189 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:18:52.189 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:18:52.189 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:52.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.213+0000 7f133980f640 1 Processor -- start 2024-03-29T11:18:52.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.213+0000 7f133980f640 1 -- start start 2024-03-29T11:18:52.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.213+0000 7f133980f640 10 monclient: init 2024-03-29T11:18:52.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.214+0000 7f133980f640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:52.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.214+0000 7f133980f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:18:52.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.214+0000 7f133980f640 10 monclient(hunting): picked mon.noname-b con 0x7f13340a9ac0 addr v1:172.21.15.47:6790/0 2024-03-29T11:18:52.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.214+0000 7f133980f640 10 monclient(hunting): picked mon.noname-a con 0x7f133413ee90 addr v1:172.21.15.47:6789/0 2024-03-29T11:18:52.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.214+0000 7f133980f640 10 monclient(hunting): picked mon.noname-c con 0x7f13340af900 addr v1:172.21.15.121:6789/0 2024-03-29T11:18:52.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.214+0000 7f133980f640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13340afd10 con 0x7f133413ee90 2024-03-29T11:18:52.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.214+0000 7f133980f640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13340af6f0 con 0x7f13340af900 2024-03-29T11:18:52.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.214+0000 7f133980f640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13340aac80 con 0x7f13340a9ac0 2024-03-29T11:18:52.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.214+0000 7f133980f640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:52.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.214+0000 7f133980f640 10 monclient(hunting): authenticate will time out at 1175.402392s 2024-03-29T11:18:52.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.214+0000 7f1332ffd640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f13340a9ac0 0x7f13340adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:51752/0 (socket says 172.21.15.47:51752) 2024-03-29T11:18:52.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.214+0000 7f1332ffd640 1 -- 172.21.15.47:0/2731390644 learned_addr learned my addr 172.21.15.47:0/2731390644 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:18:52.195 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.218+0000 7f1331ffb640 1 -- 172.21.15.47:0/2731390644 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3201779394 0 0) 0x7f13340aac80 con 0x7f13340a9ac0 2024-03-29T11:18:52.195 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.218+0000 7f1331ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:18:52.195 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.218+0000 7f1331ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:52.195 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.218+0000 7f1331ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:52.195 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.218+0000 7f1331ffb640 10 monclient(hunting): my global_id is 4466 2024-03-29T11:18:52.196 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.218+0000 7f1331ffb640 1 -- 172.21.15.47:0/2731390644 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1320003520 con 0x7f13340a9ac0 2024-03-29T11:18:52.200 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.219+0000 7f1331ffb640 1 -- 172.21.15.47:0/2731390644 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2031168071 0 0) 0x7f13340af6f0 con 0x7f13340af900 2024-03-29T11:18:52.200 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.219+0000 7f1331ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:18:52.200 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.219+0000 7f1331ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:52.200 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.219+0000 7f1331ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:52.200 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.219+0000 7f1331ffb640 10 monclient(hunting): my global_id is 4420 2024-03-29T11:18:52.200 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.219+0000 7f1331ffb640 1 -- 172.21.15.47:0/2731390644 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13340aac80 con 0x7f13340af900 2024-03-29T11:18:52.200 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.219+0000 7f1331ffb640 1 -- 172.21.15.47:0/2731390644 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3776769843 0 0) 0x7f13340afd10 con 0x7f133413ee90 2024-03-29T11:18:52.200 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.219+0000 7f1331ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:18:52.200 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.219+0000 7f1331ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:52.200 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.219+0000 7f1331ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:52.200 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.219+0000 7f1331ffb640 10 monclient(hunting): my global_id is 4419 2024-03-29T11:18:52.200 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.219+0000 7f1331ffb640 1 -- 172.21.15.47:0/2731390644 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13340af6f0 con 0x7f133413ee90 2024-03-29T11:18:52.201 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.219+0000 7f1331ffb640 1 -- 172.21.15.47:0/2731390644 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 884366195 0 0) 0x7f1320003520 con 0x7f13340a9ac0 2024-03-29T11:18:52.201 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.219+0000 7f1331ffb640 1 -- 172.21.15.47:0/2731390644 >> v1:172.21.15.121:6789/0 conn(0x7f13340af900 legacy=0x7f1334149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:52.201 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.219+0000 7f1331ffb640 1 -- 172.21.15.47:0/2731390644 >> v1:172.21.15.47:6789/0 conn(0x7f133413ee90 legacy=0x7f133413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:52.201 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.220+0000 7f1331ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:18:52.201 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.220+0000 7f1331ffb640 1 monclient: found mon.noname-b 2024-03-29T11:18:52.201 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.220+0000 7f1331ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:52.201 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.220+0000 7f1331ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-29T11:18:52.201 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.220+0000 7f1331ffb640 1 -- 172.21.15.47:0/2731390644 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13340aaf60 con 0x7f13340a9ac0 2024-03-29T11:18:52.201 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.220+0000 7f1331ffb640 10 monclient: _finish_auth 0 2024-03-29T11:18:52.201 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.220+0000 7f1331ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:18:52.201 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.220+0000 7f1331ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:52.201 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.220+0000 7f1331ffb640 1 -- 172.21.15.47:0/2731390644 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f1324001690 con 0x7f13340a9ac0 2024-03-29T11:18:52.201 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.220+0000 7f1331ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:52.201 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.220+0000 7f1331ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:18:52.202 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.220+0000 7f1331ffb640 10 monclient: dump: 2024-03-29T11:18:52.202 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:18:52.202 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:18:52.202 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:52.202 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:52.202 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:52.202 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:52.202 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:18:52.202 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:18:52.202 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:18:52.202 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:52.202 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.220+0000 7f133980f640 5 monclient: authenticate success, global_id 4466 2024-03-29T11:18:52.202 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.220+0000 7f133980f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:52.202 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.220+0000 7f1331ffb640 1 -- 172.21.15.47:0/2731390644 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f13240017f0 con 0x7f13340a9ac0 2024-03-29T11:18:52.203 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.220+0000 7f1331ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:52.203 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.220+0000 7f1331ffb640 1 -- 172.21.15.47:0/2731390644 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f1324001c00 con 0x7f13340a9ac0 2024-03-29T11:18:52.203 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.220+0000 7f1331ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:52.203 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.220+0000 7f1331ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:52.203 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.220+0000 7f1331ffb640 10 monclient: dump: 2024-03-29T11:18:52.203 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:18:52.203 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:18:52.203 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:52.203 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:52.203 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:52.203 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:52.203 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:18:52.203 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:18:52.203 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:18:52.203 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:52.204 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.220+0000 7f133980f640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:52.204 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.220+0000 7f133980f640 4 set_mon_vals no callback set 2024-03-29T11:18:52.204 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.221+0000 7f133980f640 10 monclient: shutdown 2024-03-29T11:18:52.204 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.221+0000 7f133980f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:52.204 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.221+0000 7f133980f640 1 -- 172.21.15.47:0/2731390644 >> v1:172.21.15.47:6790/0 conn(0x7f13340a9ac0 legacy=0x7f13340adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:52.204 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.222+0000 7f133980f640 1 -- 172.21.15.47:0/2731390644 shutdown_connections 2024-03-29T11:18:52.204 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.222+0000 7f133980f640 1 -- 172.21.15.47:0/2731390644 >> 172.21.15.47:0/2731390644 conn(0x7f13340b35e0 msgr2=0x7f13340b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:52.204 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.222+0000 7f133980f640 1 -- 172.21.15.47:0/2731390644 shutdown_connections 2024-03-29T11:18:52.204 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.222+0000 7f133980f640 1 -- 172.21.15.47:0/2731390644 wait complete. 2024-03-29T11:18:52.210 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.223+0000 7f133980f640 10 monclient: build_initial_monmap 2024-03-29T11:18:52.210 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.223+0000 7f133980f640 10 monclient: monmap: 2024-03-29T11:18:52.210 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:18:52.210 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:52.210 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:18:52.210 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:18:52.210 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:52.210 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:52.210 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:18:52.210 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:18:52.211 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:18:52.211 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:52.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.223+0000 7f133980f640 1 librados: starting msgr at 2024-03-29T11:18:52.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.223+0000 7f133980f640 1 librados: starting objecter 2024-03-29T11:18:52.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.223+0000 7f133980f640 1 Processor -- start 2024-03-29T11:18:52.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.224+0000 7f133980f640 1 -- start start 2024-03-29T11:18:52.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.224+0000 7f133980f640 1 librados: setting wanted keys 2024-03-29T11:18:52.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.224+0000 7f133980f640 1 librados: calling monclient init 2024-03-29T11:18:52.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.224+0000 7f133980f640 10 monclient: init 2024-03-29T11:18:52.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.224+0000 7f133980f640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:52.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.224+0000 7f133980f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:18:52.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.224+0000 7f133980f640 10 monclient(hunting): picked mon.noname-b con 0x7f13340a9ac0 addr v1:172.21.15.121:6789/0 2024-03-29T11:18:52.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.225+0000 7f133980f640 10 monclient(hunting): picked mon.noname-a con 0x7f13340af900 addr v1:172.21.15.47:6789/0 2024-03-29T11:18:52.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.225+0000 7f133980f640 10 monclient(hunting): picked mon.noname-c con 0x7f133413ee90 addr v1:172.21.15.47:6790/0 2024-03-29T11:18:52.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.225+0000 7f133980f640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f133411d4c0 con 0x7f13340af900 2024-03-29T11:18:52.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.225+0000 7f133980f640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f133411d1e0 con 0x7f13340a9ac0 2024-03-29T11:18:52.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.225+0000 7f133980f640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f133411e060 con 0x7f133413ee90 2024-03-29T11:18:52.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.225+0000 7f133980f640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:52.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.225+0000 7f133980f640 10 monclient(hunting): authenticate will time out at 1175.413906s 2024-03-29T11:18:52.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.228+0000 7f13337fe640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f133413ee90 0x7f13341dbfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:56680/0 (socket says 172.21.15.47:56680) 2024-03-29T11:18:52.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.228+0000 7f13337fe640 1 -- 172.21.15.47:0/2941308304 learned_addr learned my addr 172.21.15.47:0/2941308304 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:18:52.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.228+0000 7f132bfff640 10 client.?.objecter ms_handle_connect 0x7f133413ee90 2024-03-29T11:18:52.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.228+0000 7f132bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:52.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.228+0000 7f132bfff640 1 -- 172.21.15.47:0/2941308304 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3185226498 0 0) 0x7f133411e060 con 0x7f133413ee90 2024-03-29T11:18:52.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.228+0000 7f132bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:18:52.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.228+0000 7f132bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:52.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.228+0000 7f132bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:52.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.228+0000 7f132bfff640 10 monclient(hunting): my global_id is 4469 2024-03-29T11:18:52.213 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.228+0000 7f132bfff640 1 -- 172.21.15.47:0/2941308304 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1314004eb0 con 0x7f133413ee90 2024-03-29T11:18:52.213 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.228+0000 7f132bfff640 10 client.?.objecter ms_handle_connect 0x7f13340a9ac0 2024-03-29T11:18:52.213 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.228+0000 7f132bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:52.213 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.228+0000 7f132bfff640 10 client.?.objecter ms_handle_connect 0x7f13340af900 2024-03-29T11:18:52.213 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.228+0000 7f132bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:52.213 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.229+0000 7f132bfff640 1 -- 172.21.15.47:0/2941308304 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 196573694 0 0) 0x7f1314004eb0 con 0x7f133413ee90 2024-03-29T11:18:52.213 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.229+0000 7f132bfff640 1 -- 172.21.15.47:0/2941308304 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f133411e060 con 0x7f133413ee90 2024-03-29T11:18:52.213 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.229+0000 7f132bfff640 1 -- 172.21.15.47:0/2941308304 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f132c001d10 con 0x7f133413ee90 2024-03-29T11:18:52.213 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.229+0000 7f132bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:52.213 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.229+0000 7f132bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:18:52.213 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.229+0000 7f132bfff640 10 monclient(hunting): dump: 2024-03-29T11:18:52.213 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:18:52.213 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:18:52.213 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:52.214 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:52.214 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:52.214 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:52.214 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:18:52.214 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:18:52.214 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:18:52.214 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:52.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.229+0000 7f132bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:52.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.230+0000 7f132bfff640 1 -- 172.21.15.47:0/2941308304 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 812245225 0 0) 0x7f133411e060 con 0x7f133413ee90 2024-03-29T11:18:52.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.230+0000 7f132bfff640 1 -- 172.21.15.47:0/2941308304 >> v1:172.21.15.121:6789/0 conn(0x7f13340a9ac0 legacy=0x7f133411d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:52.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.230+0000 7f132bfff640 1 -- 172.21.15.47:0/2941308304 >> v1:172.21.15.47:6789/0 conn(0x7f13340af900 legacy=0x7f13341db8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:52.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.230+0000 7f132bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:18:52.214 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.230+0000 7f132bfff640 1 monclient: found mon.c 2024-03-29T11:18:52.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.230+0000 7f132bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:52.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.230+0000 7f132bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:18:52.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.230+0000 7f132bfff640 1 -- 172.21.15.47:0/2941308304 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13341dc6c0 con 0x7f133413ee90 2024-03-29T11:18:52.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.230+0000 7f132bfff640 10 monclient: _finish_auth 0 2024-03-29T11:18:52.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.230+0000 7f132bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:18:52.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.230+0000 7f132bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:52.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.231+0000 7f132bfff640 1 -- 172.21.15.47:0/2941308304 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f132c003990 con 0x7f133413ee90 2024-03-29T11:18:52.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.231+0000 7f132bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:52.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.231+0000 7f132bfff640 1 -- 172.21.15.47:0/2941308304 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f132c003da0 con 0x7f133413ee90 2024-03-29T11:18:52.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.231+0000 7f132bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:52.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.231+0000 7f132bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:52.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.231+0000 7f132bfff640 10 monclient: dump: 2024-03-29T11:18:52.215 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:18:52.215 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:18:52.215 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:52.216 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:52.216 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:52.216 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:52.216 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:18:52.216 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:18:52.216 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:18:52.216 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:52.216 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.232+0000 7f133980f640 5 monclient: authenticate success, global_id 4469 2024-03-29T11:18:52.216 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.232+0000 7f133980f640 10 monclient: _renew_subs 2024-03-29T11:18:52.216 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.232+0000 7f133980f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:18:52.216 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.232+0000 7f133980f640 1 -- 172.21.15.47:0/2941308304 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f133411f050 con 0x7f133413ee90 2024-03-29T11:18:52.216 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.232+0000 7f133980f640 10 client.4469.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:52.216 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.232+0000 7f133980f640 10 monclient: _renew_subs 2024-03-29T11:18:52.216 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.232+0000 7f133980f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:18:52.217 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.232+0000 7f133980f640 1 -- 172.21.15.47:0/2941308304 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f133411f440 con 0x7f133413ee90 2024-03-29T11:18:52.217 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.232+0000 7f133980f640 1 librados: init done 2024-03-29T11:18:52.217 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.232+0000 7f13317fa640 4 set_mon_vals no callback set 2024-03-29T11:18:52.217 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.233+0000 7f133980f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:52.217 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.233+0000 7f133980f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:52.218 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.233+0000 7f133980f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:18:52.218 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.233+0000 7f133980f640 1 -- 172.21.15.47:0/2941308304 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1318005080 con 0x7f133413ee90 2024-03-29T11:18:52.218 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.235+0000 7f132bfff640 1 -- 172.21.15.47:0/2941308304 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7f132c003f80 con 0x7f133413ee90 2024-03-29T11:18:52.218 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.235+0000 7f132bfff640 1 -- 172.21.15.47:0/2941308304 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7f132c084eb0 con 0x7f133413ee90 2024-03-29T11:18:52.218 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.235+0000 7f132bfff640 10 client.4469.objecter ms_dispatch 0x7f1334000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:18:52.218 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.235+0000 7f132bfff640 3 client.4469.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:18:52.218 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.235+0000 7f132bfff640 3 client.4469.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:18:52.218 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.235+0000 7f132bfff640 20 client.4469.objecter dump_active .. 0 homeless 2024-03-29T11:18:52.218 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.236+0000 7f1332ffd640 10 monclient: get_auth_request con 0x7f131406dce0 auth_method 0 2024-03-29T11:18:52.218 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.237+0000 7f132bfff640 10 client.4469.objecter ms_handle_connect 0x7f131406dce0 2024-03-29T11:18:52.220 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.243+0000 7f132bfff640 1 -- 172.21.15.47:0/2941308304 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f132c050790 con 0x7f133413ee90 2024-03-29T11:18:52.220 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.243+0000 7f132bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:52.221 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.243+0000 7f132bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:52.563 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.586+0000 7f133980f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:52.563 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.586+0000 7f133980f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:52.563 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.586+0000 7f133980f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:18:52.563 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.586+0000 7f133980f640 1 -- 172.21.15.47:0/2941308304 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1318005330 con 0x7f133413ee90 2024-03-29T11:18:52.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.587+0000 7f132bfff640 1 -- 172.21.15.47:0/2941308304 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7f132c0526c0 con 0x7f133413ee90 2024-03-29T11:18:52.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.587+0000 7f132bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:52.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.587+0000 7f132bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:52.564 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:18:52.565 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:18:52.569 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.591+0000 7f133980f640 10 librados: watch_flush enter 2024-03-29T11:18:52.569 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.592+0000 7f133980f640 10 librados: watch_flush exit 2024-03-29T11:18:52.569 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.592+0000 7f133980f640 20 client.4469.objecter shutdown clearing up homeless session... 2024-03-29T11:18:52.569 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.592+0000 7f133980f640 10 client.4469.objecter successfully canceled tick 2024-03-29T11:18:52.569 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.592+0000 7f133980f640 1 -- 172.21.15.47:0/2941308304 >> v1:172.21.15.121:6816/1964117370 conn(0x7f131406dce0 legacy=0x7f13140701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:52.569 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.592+0000 7f133980f640 10 monclient: shutdown 2024-03-29T11:18:52.569 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.592+0000 7f133980f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:52.569 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.592+0000 7f133980f640 1 -- 172.21.15.47:0/2941308304 >> v1:172.21.15.47:6790/0 conn(0x7f133413ee90 legacy=0x7f13341dbfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:52.569 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.592+0000 7f133980f640 1 -- 172.21.15.47:0/2941308304 shutdown_connections 2024-03-29T11:18:52.569 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.592+0000 7f133980f640 1 -- 172.21.15.47:0/2941308304 >> 172.21.15.47:0/2941308304 conn(0x7f13340b35e0 msgr2=0x7f133413af70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:52.569 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.592+0000 7f133980f640 1 -- 172.21.15.47:0/2941308304 shutdown_connections 2024-03-29T11:18:52.570 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.592+0000 7f133980f640 1 -- 172.21.15.47:0/2941308304 wait complete. 2024-03-29T11:18:52.570 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:52.592+0000 7f133980f640 1 librados: shutdown 2024-03-29T11:18:52.587 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:18:59.394 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:18:59.541 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.563+0000 7fca15640640 10 monclient: get_monmap_and_config 2024-03-29T11:18:59.541 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.563+0000 7fca15640640 10 monclient: build_initial_monmap 2024-03-29T11:18:59.541 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.563+0000 7fca15640640 10 monclient: monmap: 2024-03-29T11:18:59.541 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:18:59.541 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:18:59.541 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:18:59.565100+0000 2024-03-29T11:18:59.542 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:18:59.565100+0000 2024-03-29T11:18:59.542 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:59.542 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:59.542 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:18:59.542 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:18:59.542 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:18:59.542 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:59.542 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.563+0000 7fca15640640 1 Processor -- start 2024-03-29T11:18:59.542 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.563+0000 7fca15640640 1 -- start start 2024-03-29T11:18:59.542 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.563+0000 7fca15640640 10 monclient: init 2024-03-29T11:18:59.543 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.564+0000 7fca15640640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:59.543 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.564+0000 7fca15640640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:18:59.543 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.564+0000 7fca15640640 10 monclient(hunting): picked mon.noname-b con 0x7fca101457f0 addr v1:172.21.15.47:6790/0 2024-03-29T11:18:59.543 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.564+0000 7fca15640640 10 monclient(hunting): picked mon.noname-a con 0x7fca1013f650 addr v1:172.21.15.47:6789/0 2024-03-29T11:18:59.543 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.564+0000 7fca15640640 10 monclient(hunting): picked mon.noname-c con 0x7fca100a9d70 addr v1:172.21.15.121:6789/0 2024-03-29T11:18:59.543 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.564+0000 7fca15640640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca100aaef0 con 0x7fca1013f650 2024-03-29T11:18:59.543 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.564+0000 7fca15640640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca10145bd0 con 0x7fca100a9d70 2024-03-29T11:18:59.544 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.564+0000 7fca15640640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca10145eb0 con 0x7fca101457f0 2024-03-29T11:18:59.544 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.564+0000 7fca15640640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:59.544 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.564+0000 7fca15640640 10 monclient(hunting): authenticate will time out at 1182.753543s 2024-03-29T11:18:59.544 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.564+0000 7fca0effd640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fca101457f0 0x7fca1013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:44504/0 (socket says 172.21.15.47:44504) 2024-03-29T11:18:59.544 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.564+0000 7fca0effd640 1 -- 172.21.15.47:0/3872512058 learned_addr learned my addr 172.21.15.47:0/3872512058 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:18:59.545 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.565+0000 7fca0dffb640 1 -- 172.21.15.47:0/3872512058 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1173451118 0 0) 0x7fca10145eb0 con 0x7fca101457f0 2024-03-29T11:18:59.545 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.565+0000 7fca0dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:18:59.545 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.565+0000 7fca0dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:59.545 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.565+0000 7fca0dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:59.545 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.565+0000 7fca0dffb640 10 monclient(hunting): my global_id is 4472 2024-03-29T11:18:59.545 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.565+0000 7fca0dffb640 1 -- 172.21.15.47:0/3872512058 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9f8003520 con 0x7fca101457f0 2024-03-29T11:18:59.545 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.566+0000 7fca0dffb640 1 -- 172.21.15.47:0/3872512058 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 334129311 0 0) 0x7fc9f8003520 con 0x7fca101457f0 2024-03-29T11:18:59.546 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.566+0000 7fca0dffb640 1 -- 172.21.15.47:0/3872512058 >> v1:172.21.15.121:6789/0 conn(0x7fca100a9d70 legacy=0x7fca1014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:59.546 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.566+0000 7fca0dffb640 1 -- 172.21.15.47:0/3872512058 >> v1:172.21.15.47:6789/0 conn(0x7fca1013f650 legacy=0x7fca100b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:18:59.547 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.568+0000 7fca0dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:18:59.547 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.568+0000 7fca0dffb640 1 monclient: found mon.noname-b 2024-03-29T11:18:59.547 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.568+0000 7fca0dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:59.547 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.568+0000 7fca0dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-29T11:18:59.547 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.568+0000 7fca0dffb640 1 -- 172.21.15.47:0/3872512058 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca10146340 con 0x7fca101457f0 2024-03-29T11:18:59.547 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.568+0000 7fca0dffb640 10 monclient: _finish_auth 0 2024-03-29T11:18:59.547 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.568+0000 7fca0dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:18:59.547 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.568+0000 7fca0dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:59.548 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.568+0000 7fca0dffb640 1 -- 172.21.15.47:0/3872512058 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fca00001690 con 0x7fca101457f0 2024-03-29T11:18:59.548 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.568+0000 7fca0dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:59.548 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.568+0000 7fca0dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:18:59.548 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.568+0000 7fca0dffb640 10 monclient: dump: 2024-03-29T11:18:59.548 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:18:59.548 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:18:59.548 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:59.548 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:59.548 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:59.548 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:59.548 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:18:59.548 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:18:59.548 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:18:59.548 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:59.548 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.568+0000 7fca15640640 5 monclient: authenticate success, global_id 4472 2024-03-29T11:18:59.549 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.568+0000 7fca15640640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:59.549 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.569+0000 7fca0dffb640 1 -- 172.21.15.47:0/3872512058 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca00001920 con 0x7fca101457f0 2024-03-29T11:18:59.549 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.569+0000 7fca0dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:59.549 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.569+0000 7fca0dffb640 1 -- 172.21.15.47:0/3872512058 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fca00001cb0 con 0x7fca101457f0 2024-03-29T11:18:59.549 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.569+0000 7fca0dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:59.549 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.569+0000 7fca0dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:59.549 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.569+0000 7fca0dffb640 10 monclient: dump: 2024-03-29T11:18:59.549 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:18:59.549 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:18:59.549 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:59.549 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:59.549 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:59.549 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:59.549 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:18:59.549 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:18:59.550 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:18:59.550 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:59.550 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.569+0000 7fca15640640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:59.550 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.569+0000 7fca15640640 4 set_mon_vals no callback set 2024-03-29T11:18:59.553 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.570+0000 7fca15640640 10 monclient: shutdown 2024-03-29T11:18:59.553 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.570+0000 7fca15640640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:59.553 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.570+0000 7fca15640640 1 -- 172.21.15.47:0/3872512058 >> v1:172.21.15.47:6790/0 conn(0x7fca101457f0 legacy=0x7fca1013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:59.553 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.570+0000 7fca15640640 1 -- 172.21.15.47:0/3872512058 shutdown_connections 2024-03-29T11:18:59.554 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.570+0000 7fca15640640 1 -- 172.21.15.47:0/3872512058 >> 172.21.15.47:0/3872512058 conn(0x7fca10066a00 msgr2=0x7fca10066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:59.554 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.570+0000 7fca15640640 1 -- 172.21.15.47:0/3872512058 shutdown_connections 2024-03-29T11:18:59.554 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.570+0000 7fca15640640 1 -- 172.21.15.47:0/3872512058 wait complete. 2024-03-29T11:18:59.554 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.571+0000 7fca15640640 10 monclient: build_initial_monmap 2024-03-29T11:18:59.554 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.571+0000 7fca15640640 10 monclient: monmap: 2024-03-29T11:18:59.554 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:18:59.554 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:59.554 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:18:59.554 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:18:59.554 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:59.554 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:59.554 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:18:59.554 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:18:59.554 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:18:59.555 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:59.555 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.571+0000 7fca15640640 1 librados: starting msgr at 2024-03-29T11:18:59.555 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.571+0000 7fca15640640 1 librados: starting objecter 2024-03-29T11:18:59.555 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.572+0000 7fca15640640 1 Processor -- start 2024-03-29T11:18:59.555 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.572+0000 7fca15640640 1 -- start start 2024-03-29T11:18:59.555 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.572+0000 7fca15640640 1 librados: setting wanted keys 2024-03-29T11:18:59.555 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.572+0000 7fca15640640 1 librados: calling monclient init 2024-03-29T11:18:59.555 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.572+0000 7fca15640640 10 monclient: init 2024-03-29T11:18:59.555 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.573+0000 7fca15640640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:59.555 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.573+0000 7fca15640640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:18:59.555 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.574+0000 7fca15640640 10 monclient(hunting): picked mon.noname-a con 0x7fca100a9d70 addr v1:172.21.15.47:6789/0 2024-03-29T11:18:59.555 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.574+0000 7fca15640640 10 monclient(hunting): picked mon.noname-c con 0x7fca1013f650 addr v1:172.21.15.47:6790/0 2024-03-29T11:18:59.555 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.574+0000 7fca15640640 10 monclient(hunting): picked mon.noname-b con 0x7fca101457f0 addr v1:172.21.15.121:6789/0 2024-03-29T11:18:59.555 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.574+0000 7fca15640640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca10139d60 con 0x7fca100a9d70 2024-03-29T11:18:59.556 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.574+0000 7fca15640640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca101ceac0 con 0x7fca101457f0 2024-03-29T11:18:59.556 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.574+0000 7fca15640640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca101ced30 con 0x7fca1013f650 2024-03-29T11:18:59.556 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.574+0000 7fca15640640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:59.556 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.574+0000 7fca15640640 10 monclient(hunting): authenticate will time out at 1182.763701s 2024-03-29T11:18:59.556 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.574+0000 7fca0e7fc640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fca1013f650 0x7fca101cd4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:46450/0 (socket says 172.21.15.47:46450) 2024-03-29T11:18:59.556 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.574+0000 7fca0e7fc640 1 -- 172.21.15.47:0/2624176348 learned_addr learned my addr 172.21.15.47:0/2624176348 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:18:59.556 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.574+0000 7fc9f77fe640 10 client.?.objecter ms_handle_connect 0x7fca1013f650 2024-03-29T11:18:59.556 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.574+0000 7fc9f77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:59.556 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.575+0000 7fc9f77fe640 1 -- 172.21.15.47:0/2624176348 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4093979105 0 0) 0x7fca101ced30 con 0x7fca1013f650 2024-03-29T11:18:59.556 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.575+0000 7fc9f77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:18:59.556 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.575+0000 7fc9f77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:59.556 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.575+0000 7fc9f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:59.556 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.575+0000 7fc9f77fe640 10 monclient(hunting): my global_id is 4475 2024-03-29T11:18:59.556 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.575+0000 7fc9f77fe640 1 -- 172.21.15.47:0/2624176348 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9fc004eb0 con 0x7fca1013f650 2024-03-29T11:18:59.556 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.575+0000 7fc9f77fe640 10 client.?.objecter ms_handle_connect 0x7fca101457f0 2024-03-29T11:18:59.557 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.575+0000 7fc9f77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:59.557 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.575+0000 7fc9f77fe640 1 -- 172.21.15.47:0/2624176348 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4053477453 0 0) 0x7fc9fc004eb0 con 0x7fca1013f650 2024-03-29T11:18:59.557 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.575+0000 7fc9f77fe640 1 -- 172.21.15.47:0/2624176348 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca101ced30 con 0x7fca1013f650 2024-03-29T11:18:59.557 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.575+0000 7fc9f77fe640 1 -- 172.21.15.47:0/2624176348 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fca04001960 con 0x7fca1013f650 2024-03-29T11:18:59.557 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.575+0000 7fc9f77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:59.557 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.575+0000 7fc9f77fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:18:59.557 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.575+0000 7fc9f77fe640 10 monclient(hunting): dump: 2024-03-29T11:18:59.557 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:18:59.557 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:18:59.557 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:59.557 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:59.557 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:59.557 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:59.557 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:18:59.557 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:18:59.558 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:18:59.558 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:59.558 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.575+0000 7fc9f77fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:59.558 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.575+0000 7fc9f77fe640 1 -- 172.21.15.47:0/2624176348 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1973222617 0 0) 0x7fca101ceac0 con 0x7fca101457f0 2024-03-29T11:18:59.558 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.575+0000 7fc9f77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:18:59.558 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.575+0000 7fc9f77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:59.558 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.575+0000 7fc9f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:59.558 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.575+0000 7fc9f77fe640 10 monclient(hunting): my global_id is 4429 2024-03-29T11:18:59.558 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.575+0000 7fc9f77fe640 1 -- 172.21.15.47:0/2624176348 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9fc004eb0 con 0x7fca101457f0 2024-03-29T11:18:59.558 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.575+0000 7fc9f77fe640 1 -- 172.21.15.47:0/2624176348 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1028755174 0 0) 0x7fca101ced30 con 0x7fca1013f650 2024-03-29T11:18:59.558 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.575+0000 7fc9f77fe640 1 -- 172.21.15.47:0/2624176348 >> v1:172.21.15.121:6789/0 conn(0x7fca101457f0 legacy=0x7fca101e2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:59.558 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.575+0000 7fc9f77fe640 1 -- 172.21.15.47:0/2624176348 >> v1:172.21.15.47:6789/0 conn(0x7fca100a9d70 legacy=0x7fca101cb880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:18:59.561 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.576+0000 7fc9f77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:18:59.561 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.576+0000 7fc9f77fe640 1 monclient: found mon.c 2024-03-29T11:18:59.561 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.576+0000 7fc9f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:59.561 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.576+0000 7fc9f77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:18:59.561 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.576+0000 7fc9f77fe640 1 -- 172.21.15.47:0/2624176348 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca101cf010 con 0x7fca1013f650 2024-03-29T11:18:59.561 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.576+0000 7fc9f77fe640 10 monclient: _finish_auth 0 2024-03-29T11:18:59.561 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.576+0000 7fc9f77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:18:59.561 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.576+0000 7fc9f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:59.561 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.576+0000 7fca15640640 5 monclient: authenticate success, global_id 4475 2024-03-29T11:18:59.562 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.576+0000 7fca15640640 10 monclient: _renew_subs 2024-03-29T11:18:59.562 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.576+0000 7fca15640640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:18:59.562 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.576+0000 7fca15640640 1 -- 172.21.15.47:0/2624176348 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca101cb340 con 0x7fca1013f650 2024-03-29T11:18:59.562 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.576+0000 7fca15640640 10 client.4475.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:59.562 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.576+0000 7fca15640640 10 monclient: _renew_subs 2024-03-29T11:18:59.562 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.576+0000 7fca15640640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:18:59.562 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.576+0000 7fca15640640 1 -- 172.21.15.47:0/2624176348 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fca101cc170 con 0x7fca1013f650 2024-03-29T11:18:59.562 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.576+0000 7fca15640640 1 librados: init done 2024-03-29T11:18:59.562 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.577+0000 7fc9f77fe640 1 -- 172.21.15.47:0/2624176348 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca04001cc0 con 0x7fca1013f650 2024-03-29T11:18:59.562 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.577+0000 7fc9f77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:59.562 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.577+0000 7fc9f77fe640 1 -- 172.21.15.47:0/2624176348 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fca04003a40 con 0x7fca1013f650 2024-03-29T11:18:59.562 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.577+0000 7fc9f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:59.562 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.577+0000 7fc9f77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:59.562 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.577+0000 7fc9f77fe640 10 monclient: dump: 2024-03-29T11:18:59.563 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:18:59.563 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:18:59.563 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:59.563 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:18:59.563 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:59.563 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:18:59.563 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:18:59.563 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:18:59.563 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:18:59.563 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:18:59.563 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.577+0000 7fca0d7fa640 4 set_mon_vals no callback set 2024-03-29T11:18:59.563 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.578+0000 7fca15640640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:59.563 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.578+0000 7fca15640640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:59.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.578+0000 7fca15640640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:18:59.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.578+0000 7fca15640640 1 -- 172.21.15.47:0/2624176348 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca10075f10 con 0x7fca1013f650 2024-03-29T11:18:59.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.579+0000 7fc9f77fe640 1 -- 172.21.15.47:0/2624176348 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7fca04003ba0 con 0x7fca1013f650 2024-03-29T11:18:59.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.580+0000 7fc9f77fe640 1 -- 172.21.15.47:0/2624176348 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7fca04084b30 con 0x7fca1013f650 2024-03-29T11:18:59.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.580+0000 7fc9f77fe640 10 client.4475.objecter ms_dispatch 0x7fca10000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:18:59.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.580+0000 7fc9f77fe640 3 client.4475.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:18:59.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.580+0000 7fc9f77fe640 3 client.4475.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:18:59.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.580+0000 7fca0effd640 10 monclient: get_auth_request con 0x7fc9fc06dda0 auth_method 0 2024-03-29T11:18:59.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.580+0000 7fc9f77fe640 20 client.4475.objecter dump_active .. 0 homeless 2024-03-29T11:18:59.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.580+0000 7fc9f77fe640 10 client.4475.objecter ms_handle_connect 0x7fc9fc06dda0 2024-03-29T11:18:59.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.588+0000 7fc9f77fe640 1 -- 172.21.15.47:0/2624176348 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fca04050410 con 0x7fca1013f650 2024-03-29T11:18:59.566 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.588+0000 7fc9f77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:59.566 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.588+0000 7fc9f77fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:59.918 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.940+0000 7fca15640640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:59.918 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.940+0000 7fca15640640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:59.918 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.940+0000 7fca15640640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:18:59.918 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.940+0000 7fca15640640 1 -- 172.21.15.47:0/2624176348 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fca100b74d0 con 0x7fca1013f650 2024-03-29T11:18:59.919 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.941+0000 7fc9f77fe640 1 -- 172.21.15.47:0/2624176348 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7fca04052340 con 0x7fca1013f650 2024-03-29T11:18:59.919 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.941+0000 7fc9f77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:59.919 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.941+0000 7fc9f77fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:59.919 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:18:59.919 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:18:59.923 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.945+0000 7fca15640640 10 librados: watch_flush enter 2024-03-29T11:18:59.923 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.945+0000 7fca15640640 10 librados: watch_flush exit 2024-03-29T11:18:59.923 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.945+0000 7fca15640640 20 client.4475.objecter shutdown clearing up homeless session... 2024-03-29T11:18:59.923 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.945+0000 7fca15640640 10 client.4475.objecter successfully canceled tick 2024-03-29T11:18:59.924 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.946+0000 7fca15640640 1 -- 172.21.15.47:0/2624176348 >> v1:172.21.15.121:6816/1964117370 conn(0x7fc9fc06dda0 legacy=0x7fc9fc070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:59.924 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.946+0000 7fca15640640 10 monclient: shutdown 2024-03-29T11:18:59.924 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.946+0000 7fca15640640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:59.924 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.946+0000 7fca15640640 1 -- 172.21.15.47:0/2624176348 >> v1:172.21.15.47:6790/0 conn(0x7fca1013f650 legacy=0x7fca101cd4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:59.925 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.947+0000 7fca15640640 1 -- 172.21.15.47:0/2624176348 shutdown_connections 2024-03-29T11:18:59.925 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.947+0000 7fca15640640 1 -- 172.21.15.47:0/2624176348 >> 172.21.15.47:0/2624176348 conn(0x7fca10066a00 msgr2=0x7fca100b4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:59.925 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.947+0000 7fca15640640 1 -- 172.21.15.47:0/2624176348 shutdown_connections 2024-03-29T11:18:59.925 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.947+0000 7fca15640640 1 -- 172.21.15.47:0/2624176348 wait complete. 2024-03-29T11:18:59.925 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:18:59.948+0000 7fca15640640 1 librados: shutdown 2024-03-29T11:18:59.946 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:06.754 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:06.903 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.926+0000 7fc5111d0640 10 monclient: get_monmap_and_config 2024-03-29T11:19:06.904 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.926+0000 7fc5111d0640 10 monclient: build_initial_monmap 2024-03-29T11:19:06.904 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.926+0000 7fc5111d0640 10 monclient: monmap: 2024-03-29T11:19:06.904 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:19:06.904 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:19:06.904 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:19:06.927880+0000 2024-03-29T11:19:06.904 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:19:06.927880+0000 2024-03-29T11:19:06.904 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:06.904 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:19:06.904 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:19:06.904 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:19:06.904 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:19:06.904 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:19:06.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.927+0000 7fc5111d0640 1 Processor -- start 2024-03-29T11:19:06.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.927+0000 7fc5111d0640 1 -- start start 2024-03-29T11:19:06.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.927+0000 7fc5111d0640 10 monclient: init 2024-03-29T11:19:06.910 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.929+0000 7fc5111d0640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:06.910 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.929+0000 7fc5111d0640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:06.910 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.929+0000 7fc5111d0640 10 monclient(hunting): picked mon.noname-c con 0x7fc50c1457f0 addr v1:172.21.15.121:6789/0 2024-03-29T11:19:06.910 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.929+0000 7fc5111d0640 10 monclient(hunting): picked mon.noname-a con 0x7fc50c13f650 addr v1:172.21.15.47:6789/0 2024-03-29T11:19:06.910 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.929+0000 7fc5111d0640 10 monclient(hunting): picked mon.noname-b con 0x7fc50c0a9d70 addr v1:172.21.15.47:6790/0 2024-03-29T11:19:06.910 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.929+0000 7fc50a59c640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fc50c13f650 0x7fc50c0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:59988/0 (socket says 172.21.15.47:59988) 2024-03-29T11:19:06.910 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.929+0000 7fc50ad9d640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fc50c1457f0 0x7fc50c13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.47:50444/0 (socket says 172.21.15.47:50444) 2024-03-29T11:19:06.910 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.929+0000 7fc5111d0640 1 -- 172.21.15.47:0/2845315941 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc50c0aaef0 con 0x7fc50c13f650 2024-03-29T11:19:06.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.929+0000 7fc5111d0640 1 -- 172.21.15.47:0/2845315941 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc50c145bd0 con 0x7fc50c1457f0 2024-03-29T11:19:06.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.929+0000 7fc5111d0640 1 -- 172.21.15.47:0/2845315941 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc50c145eb0 con 0x7fc50c0a9d70 2024-03-29T11:19:06.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.929+0000 7fc50a59c640 1 -- 172.21.15.47:0/2845315941 learned_addr learned my addr 172.21.15.47:0/2845315941 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:19:06.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.929+0000 7fc5111d0640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:06.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.929+0000 7fc5111d0640 10 monclient(hunting): authenticate will time out at 1190.117769s 2024-03-29T11:19:06.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.930+0000 7fc509d9b640 1 -- 172.21.15.47:0/2845315941 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2419939191 0 0) 0x7fc50c145bd0 con 0x7fc50c1457f0 2024-03-29T11:19:06.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.930+0000 7fc509d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:19:06.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.930+0000 7fc509d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:06.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.930+0000 7fc509d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:06.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.930+0000 7fc509d9b640 10 monclient(hunting): my global_id is 4432 2024-03-29T11:19:06.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.930+0000 7fc509d9b640 1 -- 172.21.15.47:0/2845315941 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4f8003520 con 0x7fc50c1457f0 2024-03-29T11:19:06.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.930+0000 7fc509d9b640 1 -- 172.21.15.47:0/2845315941 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 487889251 0 0) 0x7fc50c145eb0 con 0x7fc50c0a9d70 2024-03-29T11:19:06.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.930+0000 7fc509d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:19:06.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.930+0000 7fc509d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:06.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.930+0000 7fc509d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:06.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.930+0000 7fc509d9b640 10 monclient(hunting): my global_id is 4478 2024-03-29T11:19:06.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.930+0000 7fc509d9b640 1 -- 172.21.15.47:0/2845315941 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc50c145bd0 con 0x7fc50c0a9d70 2024-03-29T11:19:06.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.930+0000 7fc509d9b640 1 -- 172.21.15.47:0/2845315941 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3950034534 0 0) 0x7fc50c0aaef0 con 0x7fc50c13f650 2024-03-29T11:19:06.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.930+0000 7fc509d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:19:06.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.930+0000 7fc509d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:06.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.930+0000 7fc509d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:06.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.930+0000 7fc509d9b640 10 monclient(hunting): my global_id is 4425 2024-03-29T11:19:06.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.930+0000 7fc509d9b640 1 -- 172.21.15.47:0/2845315941 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc50c145eb0 con 0x7fc50c13f650 2024-03-29T11:19:06.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.931+0000 7fc509d9b640 1 -- 172.21.15.47:0/2845315941 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3842849460 0 0) 0x7fc50c145bd0 con 0x7fc50c0a9d70 2024-03-29T11:19:06.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.931+0000 7fc509d9b640 1 -- 172.21.15.47:0/2845315941 >> v1:172.21.15.121:6789/0 conn(0x7fc50c1457f0 legacy=0x7fc50c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:06.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.931+0000 7fc509d9b640 1 -- 172.21.15.47:0/2845315941 >> v1:172.21.15.47:6789/0 conn(0x7fc50c13f650 legacy=0x7fc50c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:06.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.931+0000 7fc509d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:19:06.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.931+0000 7fc509d9b640 1 monclient: found mon.noname-b 2024-03-29T11:19:06.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.931+0000 7fc509d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:06.913 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.931+0000 7fc509d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-29T11:19:06.913 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.931+0000 7fc509d9b640 1 -- 172.21.15.47:0/2845315941 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc50c146340 con 0x7fc50c0a9d70 2024-03-29T11:19:06.913 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.931+0000 7fc509d9b640 10 monclient: _finish_auth 0 2024-03-29T11:19:06.913 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.931+0000 7fc509d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:19:06.913 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.931+0000 7fc509d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:06.913 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.931+0000 7fc509d9b640 1 -- 172.21.15.47:0/2845315941 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fc4f40015a0 con 0x7fc50c0a9d70 2024-03-29T11:19:06.913 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.931+0000 7fc509d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:06.913 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.931+0000 7fc509d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:06.913 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.931+0000 7fc509d9b640 10 monclient: dump: 2024-03-29T11:19:06.913 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:19:06.913 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:19:06.913 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:06.913 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:06.913 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:06.913 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:19:06.913 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:19:06.914 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:19:06.914 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:19:06.914 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:19:06.914 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.931+0000 7fc5111d0640 5 monclient: authenticate success, global_id 4478 2024-03-29T11:19:06.914 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.931+0000 7fc5111d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:06.914 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.931+0000 7fc509d9b640 1 -- 172.21.15.47:0/2845315941 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc4f40017c0 con 0x7fc50c0a9d70 2024-03-29T11:19:06.914 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.931+0000 7fc509d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:06.914 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.931+0000 7fc509d9b640 1 -- 172.21.15.47:0/2845315941 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fc4f4001b50 con 0x7fc50c0a9d70 2024-03-29T11:19:06.914 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.931+0000 7fc5111d0640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:06.914 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.931+0000 7fc5111d0640 4 set_mon_vals no callback set 2024-03-29T11:19:06.914 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.931+0000 7fc509d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:06.914 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.932+0000 7fc509d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:06.914 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.932+0000 7fc509d9b640 10 monclient: dump: 2024-03-29T11:19:06.915 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:19:06.915 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:19:06.915 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:06.915 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:06.915 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:06.915 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:19:06.915 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:19:06.915 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:19:06.915 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:19:06.915 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:19:06.915 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.932+0000 7fc5111d0640 10 monclient: shutdown 2024-03-29T11:19:06.915 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.932+0000 7fc5111d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:06.915 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.932+0000 7fc5111d0640 1 -- 172.21.15.47:0/2845315941 >> v1:172.21.15.47:6790/0 conn(0x7fc50c0a9d70 legacy=0x7fc50c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:06.915 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.933+0000 7fc5111d0640 1 -- 172.21.15.47:0/2845315941 shutdown_connections 2024-03-29T11:19:06.915 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.933+0000 7fc5111d0640 1 -- 172.21.15.47:0/2845315941 >> 172.21.15.47:0/2845315941 conn(0x7fc50c066a00 msgr2=0x7fc50c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:06.916 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.933+0000 7fc5111d0640 1 -- 172.21.15.47:0/2845315941 shutdown_connections 2024-03-29T11:19:06.916 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.933+0000 7fc5111d0640 1 -- 172.21.15.47:0/2845315941 wait complete. 2024-03-29T11:19:06.920 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.934+0000 7fc5111d0640 10 monclient: build_initial_monmap 2024-03-29T11:19:06.920 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.934+0000 7fc5111d0640 10 monclient: monmap: 2024-03-29T11:19:06.921 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:19:06.921 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:06.921 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:19:06.921 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:19:06.921 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:06.921 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:19:06.921 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:19:06.921 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:19:06.921 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:19:06.921 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:19:06.921 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.934+0000 7fc5111d0640 1 librados: starting msgr at 2024-03-29T11:19:06.921 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.934+0000 7fc5111d0640 1 librados: starting objecter 2024-03-29T11:19:06.921 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.935+0000 7fc5111d0640 1 Processor -- start 2024-03-29T11:19:06.922 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.935+0000 7fc5111d0640 1 -- start start 2024-03-29T11:19:06.922 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.935+0000 7fc5111d0640 1 librados: setting wanted keys 2024-03-29T11:19:06.922 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.935+0000 7fc5111d0640 1 librados: calling monclient init 2024-03-29T11:19:06.922 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.935+0000 7fc5111d0640 10 monclient: init 2024-03-29T11:19:06.922 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.936+0000 7fc5111d0640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:06.922 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.936+0000 7fc5111d0640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:06.922 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.936+0000 7fc5111d0640 10 monclient(hunting): picked mon.noname-a con 0x7fc50c0a9d70 addr v1:172.21.15.47:6789/0 2024-03-29T11:19:06.922 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.936+0000 7fc5111d0640 10 monclient(hunting): picked mon.noname-c con 0x7fc50c13f650 addr v1:172.21.15.47:6790/0 2024-03-29T11:19:06.922 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.936+0000 7fc5111d0640 10 monclient(hunting): picked mon.noname-b con 0x7fc50c1457f0 addr v1:172.21.15.121:6789/0 2024-03-29T11:19:06.922 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.936+0000 7fc5111d0640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc50c0ad270 con 0x7fc50c0a9d70 2024-03-29T11:19:06.922 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.936+0000 7fc5111d0640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc50c0ad550 con 0x7fc50c1457f0 2024-03-29T11:19:06.922 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.936+0000 7fc50ad9d640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fc50c0a9d70 0x7fc50c0b0e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:60340/0 (socket says 172.21.15.47:60340) 2024-03-29T11:19:06.922 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.936+0000 7fc50ad9d640 1 -- 172.21.15.47:0/3795665846 learned_addr learned my addr 172.21.15.47:0/3795665846 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:19:06.923 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.936+0000 7fc5111d0640 1 -- 172.21.15.47:0/3795665846 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc50c0acf50 con 0x7fc50c13f650 2024-03-29T11:19:06.923 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.936+0000 7fc5111d0640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:06.923 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.936+0000 7fc5111d0640 10 monclient(hunting): authenticate will time out at 1190.124629s 2024-03-29T11:19:06.923 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.936+0000 7fc5037fe640 10 client.?.objecter ms_handle_connect 0x7fc50c0a9d70 2024-03-29T11:19:06.923 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.936+0000 7fc5037fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:06.923 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.937+0000 7fc5037fe640 10 client.?.objecter ms_handle_connect 0x7fc50c13f650 2024-03-29T11:19:06.923 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.937+0000 7fc5037fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:06.923 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.937+0000 7fc5037fe640 10 client.?.objecter ms_handle_connect 0x7fc50c1457f0 2024-03-29T11:19:06.923 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.937+0000 7fc5037fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:06.923 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.937+0000 7fc5037fe640 1 -- 172.21.15.47:0/3795665846 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2972409041 0 0) 0x7fc50c0ad550 con 0x7fc50c1457f0 2024-03-29T11:19:06.923 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.937+0000 7fc5037fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:06.923 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.937+0000 7fc5037fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:06.923 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.937+0000 7fc5037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:06.923 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.937+0000 7fc5037fe640 10 monclient(hunting): my global_id is 4435 2024-03-29T11:19:06.924 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.937+0000 7fc5037fe640 1 -- 172.21.15.47:0/3795665846 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4ec004eb0 con 0x7fc50c1457f0 2024-03-29T11:19:06.924 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.937+0000 7fc5037fe640 1 -- 172.21.15.47:0/3795665846 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3168555795 0 0) 0x7fc50c0acf50 con 0x7fc50c13f650 2024-03-29T11:19:06.924 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.937+0000 7fc5037fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:06.924 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.937+0000 7fc5037fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:06.924 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.937+0000 7fc5037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:06.924 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.937+0000 7fc5037fe640 10 monclient(hunting): my global_id is 4481 2024-03-29T11:19:06.924 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.937+0000 7fc5037fe640 1 -- 172.21.15.47:0/3795665846 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc50c0ad550 con 0x7fc50c13f650 2024-03-29T11:19:06.924 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.938+0000 7fc5037fe640 1 -- 172.21.15.47:0/3795665846 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3939236092 0 0) 0x7fc50c0ad550 con 0x7fc50c13f650 2024-03-29T11:19:06.924 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.938+0000 7fc5037fe640 1 -- 172.21.15.47:0/3795665846 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc50c0acf50 con 0x7fc50c13f650 2024-03-29T11:19:06.924 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.938+0000 7fc5037fe640 1 -- 172.21.15.47:0/3795665846 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1844075845 0 0) 0x7fc4ec004eb0 con 0x7fc50c1457f0 2024-03-29T11:19:06.924 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.938+0000 7fc5037fe640 1 -- 172.21.15.47:0/3795665846 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc4ec006900 con 0x7fc50c1457f0 2024-03-29T11:19:06.924 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.938+0000 7fc5037fe640 1 -- 172.21.15.47:0/3795665846 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1991879435 0 0) 0x7fc50c0ad270 con 0x7fc50c0a9d70 2024-03-29T11:19:06.924 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.938+0000 7fc5037fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:06.924 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.938+0000 7fc5037fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:06.924 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.938+0000 7fc5037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:06.925 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.938+0000 7fc5037fe640 10 monclient(hunting): my global_id is 4428 2024-03-29T11:19:06.925 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.938+0000 7fc5037fe640 1 -- 172.21.15.47:0/3795665846 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4ec004eb0 con 0x7fc50c0a9d70 2024-03-29T11:19:06.925 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.938+0000 7fc5037fe640 1 -- 172.21.15.47:0/3795665846 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fc4fc001870 con 0x7fc50c13f650 2024-03-29T11:19:06.925 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.938+0000 7fc5037fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:06.925 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.938+0000 7fc5037fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:06.925 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.938+0000 7fc5037fe640 10 monclient(hunting): dump: 2024-03-29T11:19:06.925 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:19:06.925 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:19:06.925 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:06.925 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:06.925 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:06.925 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:19:06.925 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:19:06.925 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:19:06.925 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:19:06.925 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:19:06.926 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.938+0000 7fc5037fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:06.926 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.938+0000 7fc5037fe640 1 -- 172.21.15.47:0/3795665846 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fc4f40014e0 con 0x7fc50c1457f0 2024-03-29T11:19:06.926 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.938+0000 7fc5037fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:06.926 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.938+0000 7fc5037fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:06.926 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.938+0000 7fc5037fe640 10 monclient(hunting): dump: 2024-03-29T11:19:06.926 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:19:06.926 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:19:06.926 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:06.926 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:06.926 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:06.926 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:19:06.926 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:19:06.926 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:19:06.926 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:19:06.927 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:19:06.927 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.938+0000 7fc5037fe640 1 -- 172.21.15.47:0/3795665846 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 80024022 0 0) 0x7fc50c0acf50 con 0x7fc50c13f650 2024-03-29T11:19:06.927 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.938+0000 7fc5037fe640 1 -- 172.21.15.47:0/3795665846 >> v1:172.21.15.121:6789/0 conn(0x7fc50c1457f0 legacy=0x7fc50c1e2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:06.927 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.938+0000 7fc5037fe640 1 -- 172.21.15.47:0/3795665846 >> v1:172.21.15.47:6789/0 conn(0x7fc50c0a9d70 legacy=0x7fc50c0b0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:06.927 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.938+0000 7fc5037fe640 10 monclient: _finish_hunting 0 2024-03-29T11:19:06.927 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.938+0000 7fc5037fe640 1 monclient: found mon.c 2024-03-29T11:19:06.927 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.938+0000 7fc5037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:06.927 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.938+0000 7fc5037fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:19:06.927 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.938+0000 7fc5037fe640 1 -- 172.21.15.47:0/3795665846 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc50c0ae420 con 0x7fc50c13f650 2024-03-29T11:19:06.927 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.938+0000 7fc5037fe640 10 monclient: _finish_auth 0 2024-03-29T11:19:06.927 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.938+0000 7fc5037fe640 10 monclient: _check_auth_tickets 2024-03-29T11:19:06.927 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.938+0000 7fc5037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:06.927 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.938+0000 7fc5111d0640 5 monclient: authenticate success, global_id 4481 2024-03-29T11:19:06.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.938+0000 7fc5111d0640 10 monclient: _renew_subs 2024-03-29T11:19:06.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.938+0000 7fc5111d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:19:06.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.938+0000 7fc5111d0640 1 -- 172.21.15.47:0/3795665846 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc50c1424f0 con 0x7fc50c13f650 2024-03-29T11:19:06.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.938+0000 7fc5111d0640 10 client.4481.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:06.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.938+0000 7fc5111d0640 10 monclient: _renew_subs 2024-03-29T11:19:06.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.938+0000 7fc5111d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:19:06.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.938+0000 7fc5111d0640 1 -- 172.21.15.47:0/3795665846 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc50c0b1700 con 0x7fc50c13f650 2024-03-29T11:19:06.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.938+0000 7fc5111d0640 1 librados: init done 2024-03-29T11:19:06.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.938+0000 7fc5037fe640 1 -- 172.21.15.47:0/3795665846 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc4fc001d40 con 0x7fc50c13f650 2024-03-29T11:19:06.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.938+0000 7fc5037fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:06.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.938+0000 7fc5037fe640 1 -- 172.21.15.47:0/3795665846 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fc4fc004a00 con 0x7fc50c13f650 2024-03-29T11:19:06.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.938+0000 7fc5037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:06.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.938+0000 7fc50959a640 4 set_mon_vals no callback set 2024-03-29T11:19:06.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.938+0000 7fc5037fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:06.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.938+0000 7fc5037fe640 10 monclient: dump: 2024-03-29T11:19:06.929 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:19:06.929 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:19:06.929 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:06.929 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:06.929 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:06.929 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:19:06.929 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:19:06.929 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:19:06.929 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:19:06.929 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:19:06.929 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.939+0000 7fc5111d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:06.929 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.939+0000 7fc5111d0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:06.929 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.939+0000 7fc5111d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:19:06.929 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.939+0000 7fc5111d0640 1 -- 172.21.15.47:0/3795665846 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4f0005080 con 0x7fc50c13f650 2024-03-29T11:19:06.929 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.940+0000 7fc5037fe640 1 -- 172.21.15.47:0/3795665846 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7fc4fc004be0 con 0x7fc50c13f650 2024-03-29T11:19:06.929 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.941+0000 7fc5037fe640 1 -- 172.21.15.47:0/3795665846 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7fc4fc084850 con 0x7fc50c13f650 2024-03-29T11:19:06.930 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.941+0000 7fc5037fe640 10 client.4481.objecter ms_dispatch 0x7fc50c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:06.930 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.941+0000 7fc5037fe640 3 client.4481.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:06.930 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.941+0000 7fc5037fe640 3 client.4481.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:06.930 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.941+0000 7fc5037fe640 20 client.4481.objecter dump_active .. 0 homeless 2024-03-29T11:19:06.930 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.941+0000 7fc50ad9d640 10 monclient: get_auth_request con 0x7fc4ec06e390 auth_method 0 2024-03-29T11:19:06.930 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.942+0000 7fc5037fe640 10 client.4481.objecter ms_handle_connect 0x7fc4ec06e390 2024-03-29T11:19:06.930 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.950+0000 7fc5037fe640 1 -- 172.21.15.47:0/3795665846 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc4fc050190 con 0x7fc50c13f650 2024-03-29T11:19:06.930 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.950+0000 7fc5037fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:06.930 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:06.950+0000 7fc5037fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:07.318 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:07.341+0000 7fc5111d0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:07.318 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:07.341+0000 7fc5111d0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:07.318 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:07.341+0000 7fc5111d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:19:07.318 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:07.341+0000 7fc5111d0640 1 -- 172.21.15.47:0/3795665846 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc4f0005330 con 0x7fc50c13f650 2024-03-29T11:19:07.319 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:19:07.319 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:07.320 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:07.342+0000 7fc5037fe640 1 -- 172.21.15.47:0/3795665846 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7fc4fc083f00 con 0x7fc50c13f650 2024-03-29T11:19:07.320 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:07.342+0000 7fc5037fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:07.320 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:07.342+0000 7fc5037fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:07.322 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:07.346+0000 7fc5111d0640 10 librados: watch_flush enter 2024-03-29T11:19:07.322 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:07.346+0000 7fc5111d0640 10 librados: watch_flush exit 2024-03-29T11:19:07.323 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:07.346+0000 7fc5111d0640 20 client.4481.objecter shutdown clearing up homeless session... 2024-03-29T11:19:07.323 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:07.346+0000 7fc5111d0640 10 client.4481.objecter successfully canceled tick 2024-03-29T11:19:07.324 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:07.346+0000 7fc5111d0640 1 -- 172.21.15.47:0/3795665846 >> v1:172.21.15.121:6816/1964117370 conn(0x7fc4ec06e390 legacy=0x7fc4ec070850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:07.324 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:07.346+0000 7fc5111d0640 10 monclient: shutdown 2024-03-29T11:19:07.324 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:07.346+0000 7fc5111d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:07.324 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:07.346+0000 7fc5111d0640 1 -- 172.21.15.47:0/3795665846 >> v1:172.21.15.47:6790/0 conn(0x7fc50c13f650 legacy=0x7fc50c1da850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:07.324 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:07.346+0000 7fc5111d0640 1 -- 172.21.15.47:0/3795665846 shutdown_connections 2024-03-29T11:19:07.324 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:07.346+0000 7fc5111d0640 1 -- 172.21.15.47:0/3795665846 >> 172.21.15.47:0/3795665846 conn(0x7fc50c066a00 msgr2=0x7fc50c13ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:07.324 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:07.346+0000 7fc5111d0640 1 -- 172.21.15.47:0/3795665846 shutdown_connections 2024-03-29T11:19:07.324 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:07.346+0000 7fc5111d0640 1 -- 172.21.15.47:0/3795665846 wait complete. 2024-03-29T11:19:07.324 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:07.347+0000 7fc5111d0640 1 librados: shutdown 2024-03-29T11:19:07.339 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:14.144 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:14.323 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.346+0000 7fb5da224640 10 monclient: get_monmap_and_config 2024-03-29T11:19:14.323 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.346+0000 7fb5da224640 10 monclient: build_initial_monmap 2024-03-29T11:19:14.323 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.346+0000 7fb5da224640 10 monclient: monmap: 2024-03-29T11:19:14.323 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:19:14.323 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:19:14.324 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:19:14.347246+0000 2024-03-29T11:19:14.324 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:19:14.347246+0000 2024-03-29T11:19:14.324 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:14.324 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:19:14.324 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:19:14.324 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:19:14.324 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:19:14.324 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:19:14.324 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.347+0000 7fb5da224640 1 Processor -- start 2024-03-29T11:19:14.325 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.347+0000 7fb5da224640 1 -- start start 2024-03-29T11:19:14.325 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.347+0000 7fb5da224640 10 monclient: init 2024-03-29T11:19:14.325 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.348+0000 7fb5da224640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:14.325 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.348+0000 7fb5da224640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:19:14.330 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.352+0000 7fb5da224640 10 monclient(hunting): picked mon.noname-a con 0x7fb5cc038c60 addr v1:172.21.15.47:6789/0 2024-03-29T11:19:14.333 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.354+0000 7fb5da224640 10 monclient(hunting): picked mon.noname-c con 0x7fb5cc0324b0 addr v1:172.21.15.121:6789/0 2024-03-29T11:19:14.333 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.354+0000 7fb5d9222640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fb5cc038c60 0x7fb5cc0c4ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:44323/0 (socket says 172.21.15.47:44323) 2024-03-29T11:19:14.333 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.355+0000 7fb5d8a21640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fb5cc0324b0 0x7fb5cc0cb4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.47:39125/0 (socket says 172.21.15.47:39125) 2024-03-29T11:19:14.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.356+0000 7fb5d9222640 1 -- 172.21.15.47:0/1031107527 learned_addr learned my addr 172.21.15.47:0/1031107527 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:19:14.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.356+0000 7fb5da224640 10 monclient(hunting): picked mon.noname-b con 0x7fb5cc0d11e0 addr v1:172.21.15.47:6790/0 2024-03-29T11:19:14.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.356+0000 7fb5da224640 1 -- 172.21.15.47:0/1031107527 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5cc0caf80 con 0x7fb5cc038c60 2024-03-29T11:19:14.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.357+0000 7fb5da224640 1 -- 172.21.15.47:0/1031107527 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5cc039850 con 0x7fb5cc0324b0 2024-03-29T11:19:14.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.357+0000 7fb5da224640 1 -- 172.21.15.47:0/1031107527 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5cc0c78d0 con 0x7fb5cc0d11e0 2024-03-29T11:19:14.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.357+0000 7fb5da224640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:14.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.357+0000 7fb5da224640 10 monclient(hunting): authenticate will time out at 1197.545258s 2024-03-29T11:19:14.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.357+0000 7fb5cbfff640 1 -- 172.21.15.47:0/1031107527 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4169197145 0 0) 0x7fb5cc039850 con 0x7fb5cc0324b0 2024-03-29T11:19:14.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.357+0000 7fb5cbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:14.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.357+0000 7fb5cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:14.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.357+0000 7fb5cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:14.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.357+0000 7fb5cbfff640 10 monclient(hunting): my global_id is 4438 2024-03-29T11:19:14.338 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.357+0000 7fb5cbfff640 1 -- 172.21.15.47:0/1031107527 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5c0003520 con 0x7fb5cc0324b0 2024-03-29T11:19:14.338 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.357+0000 7fb5cbfff640 1 -- 172.21.15.47:0/1031107527 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1519194946 0 0) 0x7fb5cc0c78d0 con 0x7fb5cc0d11e0 2024-03-29T11:19:14.338 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.357+0000 7fb5cbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:14.338 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.357+0000 7fb5cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:14.338 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.357+0000 7fb5cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:14.338 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.357+0000 7fb5cbfff640 10 monclient(hunting): my global_id is 4484 2024-03-29T11:19:14.338 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.357+0000 7fb5cbfff640 1 -- 172.21.15.47:0/1031107527 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5cc039850 con 0x7fb5cc0d11e0 2024-03-29T11:19:14.338 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.358+0000 7fb5cbfff640 1 -- 172.21.15.47:0/1031107527 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3174589118 0 0) 0x7fb5c0003520 con 0x7fb5cc0324b0 2024-03-29T11:19:14.338 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.358+0000 7fb5cbfff640 1 -- 172.21.15.47:0/1031107527 >> v1:172.21.15.47:6790/0 conn(0x7fb5cc0d11e0 legacy=0x7fb5cc0d35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:14.338 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.358+0000 7fb5cbfff640 1 -- 172.21.15.47:0/1031107527 >> v1:172.21.15.47:6789/0 conn(0x7fb5cc038c60 legacy=0x7fb5cc0c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:14.338 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.358+0000 7fb5cbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:19:14.338 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.358+0000 7fb5cbfff640 1 monclient: found mon.noname-c 2024-03-29T11:19:14.338 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.358+0000 7fb5cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:14.339 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.358+0000 7fb5cbfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:19:14.339 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.358+0000 7fb5cbfff640 1 -- 172.21.15.47:0/1031107527 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5cc0c7a60 con 0x7fb5cc0324b0 2024-03-29T11:19:14.339 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.358+0000 7fb5cbfff640 10 monclient: _finish_auth 0 2024-03-29T11:19:14.339 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.358+0000 7fb5cbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:19:14.339 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.358+0000 7fb5cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:14.339 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.358+0000 7fb5cbfff640 1 -- 172.21.15.47:0/1031107527 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fb5c40015c0 con 0x7fb5cc0324b0 2024-03-29T11:19:14.339 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.359+0000 7fb5da224640 5 monclient: authenticate success, global_id 4438 2024-03-29T11:19:14.339 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.359+0000 7fb5da224640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:14.339 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.359+0000 7fb5cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:14.339 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.359+0000 7fb5cbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:14.339 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.359+0000 7fb5cbfff640 10 monclient: dump: 2024-03-29T11:19:14.339 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:19:14.339 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:19:14.339 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:14.339 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:14.340 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:14.340 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:19:14.340 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:19:14.340 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:19:14.340 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:19:14.340 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:19:14.340 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.359+0000 7fb5da224640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:14.342 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.360+0000 7fb5cbfff640 1 -- 172.21.15.47:0/1031107527 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb5c4001850 con 0x7fb5cc0324b0 2024-03-29T11:19:14.342 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.360+0000 7fb5cbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:14.342 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.360+0000 7fb5cbfff640 1 -- 172.21.15.47:0/1031107527 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fb5c4001be0 con 0x7fb5cc0324b0 2024-03-29T11:19:14.342 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.360+0000 7fb5cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:14.342 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.360+0000 7fb5cbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:14.342 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.360+0000 7fb5cbfff640 10 monclient: dump: 2024-03-29T11:19:14.342 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:19:14.342 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:19:14.342 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:14.342 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:14.343 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:14.343 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:19:14.343 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:19:14.343 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:19:14.343 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:19:14.343 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:19:14.343 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.361+0000 7fb5da224640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:14.343 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.361+0000 7fb5da224640 4 set_mon_vals no callback set 2024-03-29T11:19:14.343 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.361+0000 7fb5da224640 10 monclient: shutdown 2024-03-29T11:19:14.343 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.362+0000 7fb5da224640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:14.343 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.362+0000 7fb5da224640 1 -- 172.21.15.47:0/1031107527 >> v1:172.21.15.121:6789/0 conn(0x7fb5cc0324b0 legacy=0x7fb5cc0cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:14.343 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.362+0000 7fb5da224640 1 -- 172.21.15.47:0/1031107527 shutdown_connections 2024-03-29T11:19:14.343 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.362+0000 7fb5da224640 1 -- 172.21.15.47:0/1031107527 >> 172.21.15.47:0/1031107527 conn(0x7fb5cc005c20 msgr2=0x7fb5cc006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:14.343 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.362+0000 7fb5da224640 1 -- 172.21.15.47:0/1031107527 shutdown_connections 2024-03-29T11:19:14.343 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.362+0000 7fb5da224640 1 -- 172.21.15.47:0/1031107527 wait complete. 2024-03-29T11:19:14.344 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.363+0000 7fb5da224640 10 monclient: build_initial_monmap 2024-03-29T11:19:14.344 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.363+0000 7fb5da224640 10 monclient: monmap: 2024-03-29T11:19:14.344 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:19:14.344 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:14.344 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:19:14.344 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:19:14.344 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:14.344 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:19:14.344 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:19:14.344 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:19:14.344 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:19:14.344 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:19:14.344 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.364+0000 7fb5da224640 1 librados: starting msgr at 2024-03-29T11:19:14.344 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.364+0000 7fb5da224640 1 librados: starting objecter 2024-03-29T11:19:14.345 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.364+0000 7fb5da224640 1 Processor -- start 2024-03-29T11:19:14.345 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.364+0000 7fb5da224640 1 -- start start 2024-03-29T11:19:14.345 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.364+0000 7fb5da224640 1 librados: setting wanted keys 2024-03-29T11:19:14.345 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.364+0000 7fb5da224640 1 librados: calling monclient init 2024-03-29T11:19:14.345 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.364+0000 7fb5da224640 10 monclient: init 2024-03-29T11:19:14.346 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.365+0000 7fb5da224640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:14.346 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.365+0000 7fb5da224640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:14.346 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.368+0000 7fb5da224640 10 monclient(hunting): picked mon.noname-c con 0x7fb5cc038c60 addr v1:172.21.15.47:6790/0 2024-03-29T11:19:14.346 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.368+0000 7fb5da224640 10 monclient(hunting): picked mon.noname-b con 0x7fb5cc0d11e0 addr v1:172.21.15.121:6789/0 2024-03-29T11:19:14.346 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.368+0000 7fb5da224640 10 monclient(hunting): picked mon.noname-a con 0x7fb5cc034840 addr v1:172.21.15.47:6789/0 2024-03-29T11:19:14.346 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.368+0000 7fb5da224640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5cc035500 con 0x7fb5cc034840 2024-03-29T11:19:14.346 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.368+0000 7fb5da224640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5cc035960 con 0x7fb5cc0d11e0 2024-03-29T11:19:14.346 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.368+0000 7fb5da224640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5cc035af0 con 0x7fb5cc038c60 2024-03-29T11:19:14.346 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.368+0000 7fb5da224640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:14.346 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.368+0000 7fb5da224640 10 monclient(hunting): authenticate will time out at 1197.556718s 2024-03-29T11:19:14.349 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.372+0000 7fb5d9222640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fb5cc038c60 0x7fb5cc15ce10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:48275/0 (socket says 172.21.15.47:48275) 2024-03-29T11:19:14.349 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.372+0000 7fb5d9222640 1 -- 172.21.15.47:0/4058192102 learned_addr learned my addr 172.21.15.47:0/4058192102 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:19:14.356 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.375+0000 7fb5c9ffb640 10 client.?.objecter ms_handle_connect 0x7fb5cc0d11e0 2024-03-29T11:19:14.356 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.375+0000 7fb5c9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:14.356 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.375+0000 7fb5c9ffb640 10 client.?.objecter ms_handle_connect 0x7fb5cc038c60 2024-03-29T11:19:14.357 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.375+0000 7fb5c9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:14.357 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.375+0000 7fb5c9ffb640 10 client.?.objecter ms_handle_connect 0x7fb5cc034840 2024-03-29T11:19:14.357 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.375+0000 7fb5c9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:14.357 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.375+0000 7fb5c9ffb640 1 -- 172.21.15.47:0/4058192102 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2434002433 0 0) 0x7fb5cc035960 con 0x7fb5cc0d11e0 2024-03-29T11:19:14.357 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.375+0000 7fb5c9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:14.357 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.375+0000 7fb5c9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:14.357 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.375+0000 7fb5c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:14.357 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.375+0000 7fb5c9ffb640 10 monclient(hunting): my global_id is 4441 2024-03-29T11:19:14.357 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.375+0000 7fb5c9ffb640 1 -- 172.21.15.47:0/4058192102 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5b8004eb0 con 0x7fb5cc0d11e0 2024-03-29T11:19:14.357 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.375+0000 7fb5c9ffb640 1 -- 172.21.15.47:0/4058192102 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 966110107 0 0) 0x7fb5cc035af0 con 0x7fb5cc038c60 2024-03-29T11:19:14.357 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.375+0000 7fb5c9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:14.357 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.375+0000 7fb5c9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:14.357 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.375+0000 7fb5c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:14.357 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.375+0000 7fb5c9ffb640 10 monclient(hunting): my global_id is 4487 2024-03-29T11:19:14.358 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.375+0000 7fb5c9ffb640 1 -- 172.21.15.47:0/4058192102 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5cc035960 con 0x7fb5cc038c60 2024-03-29T11:19:14.358 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.376+0000 7fb5c9ffb640 1 -- 172.21.15.47:0/4058192102 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 610008395 0 0) 0x7fb5cc035960 con 0x7fb5cc038c60 2024-03-29T11:19:14.358 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.376+0000 7fb5c9ffb640 1 -- 172.21.15.47:0/4058192102 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb5cc035af0 con 0x7fb5cc038c60 2024-03-29T11:19:14.358 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.376+0000 7fb5c9ffb640 1 -- 172.21.15.47:0/4058192102 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1446671401 0 0) 0x7fb5b8004eb0 con 0x7fb5cc0d11e0 2024-03-29T11:19:14.358 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.376+0000 7fb5c9ffb640 1 -- 172.21.15.47:0/4058192102 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb5cc035960 con 0x7fb5cc0d11e0 2024-03-29T11:19:14.358 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.376+0000 7fb5c9ffb640 1 -- 172.21.15.47:0/4058192102 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fb5d0001830 con 0x7fb5cc038c60 2024-03-29T11:19:14.358 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.376+0000 7fb5c9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:14.358 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.376+0000 7fb5c9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:14.358 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.376+0000 7fb5c9ffb640 10 monclient(hunting): dump: 2024-03-29T11:19:14.358 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:19:14.358 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:19:14.359 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:14.359 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:14.359 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:14.359 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:19:14.359 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:19:14.359 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:19:14.359 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:19:14.359 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:19:14.359 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.376+0000 7fb5c9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:14.359 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.376+0000 7fb5c9ffb640 1 -- 172.21.15.47:0/4058192102 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fb5c4001480 con 0x7fb5cc0d11e0 2024-03-29T11:19:14.359 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.376+0000 7fb5c9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:14.359 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.376+0000 7fb5c9ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:14.359 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.376+0000 7fb5c9ffb640 10 monclient(hunting): dump: 2024-03-29T11:19:14.359 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:19:14.360 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:19:14.360 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:14.360 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:14.360 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:14.360 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:19:14.360 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:19:14.360 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:19:14.360 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:19:14.360 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:19:14.360 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.376+0000 7fb5c9ffb640 1 -- 172.21.15.47:0/4058192102 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1641724878 0 0) 0x7fb5cc035af0 con 0x7fb5cc038c60 2024-03-29T11:19:14.360 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.376+0000 7fb5c9ffb640 1 -- 172.21.15.47:0/4058192102 >> v1:172.21.15.121:6789/0 conn(0x7fb5cc0d11e0 legacy=0x7fb5cc0be380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:14.360 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.376+0000 7fb5c9ffb640 1 -- 172.21.15.47:0/4058192102 >> v1:172.21.15.47:6789/0 conn(0x7fb5cc034840 legacy=0x7fb5cc168060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:14.360 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.376+0000 7fb5c9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:19:14.360 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.376+0000 7fb5c9ffb640 1 monclient: found mon.c 2024-03-29T11:19:14.360 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.376+0000 7fb5c9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:14.361 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.376+0000 7fb5c9ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:19:14.361 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.376+0000 7fb5c9ffb640 1 -- 172.21.15.47:0/4058192102 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5cc0c56d0 con 0x7fb5cc038c60 2024-03-29T11:19:14.361 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.377+0000 7fb5c9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:19:14.361 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.377+0000 7fb5c9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:19:14.361 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.377+0000 7fb5c9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:14.361 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.377+0000 7fb5da224640 5 monclient: authenticate success, global_id 4487 2024-03-29T11:19:14.361 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.377+0000 7fb5da224640 10 monclient: _renew_subs 2024-03-29T11:19:14.361 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.377+0000 7fb5da224640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:19:14.361 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.377+0000 7fb5da224640 1 -- 172.21.15.47:0/4058192102 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5cc037010 con 0x7fb5cc038c60 2024-03-29T11:19:14.361 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.377+0000 7fb5da224640 10 client.4487.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:14.361 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.377+0000 7fb5da224640 10 monclient: _renew_subs 2024-03-29T11:19:14.361 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.377+0000 7fb5da224640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:19:14.361 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.377+0000 7fb5da224640 1 -- 172.21.15.47:0/4058192102 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5cc15d700 con 0x7fb5cc038c60 2024-03-29T11:19:14.361 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.377+0000 7fb5da224640 1 librados: init done 2024-03-29T11:19:14.362 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.378+0000 7fb5c9ffb640 1 -- 172.21.15.47:0/4058192102 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb5d00047e0 con 0x7fb5cc038c60 2024-03-29T11:19:14.362 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.378+0000 7fb5c9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:14.362 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.379+0000 7fb5caffd640 4 set_mon_vals no callback set 2024-03-29T11:19:14.362 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.380+0000 7fb5c9ffb640 1 -- 172.21.15.47:0/4058192102 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fb5d0001d80 con 0x7fb5cc038c60 2024-03-29T11:19:14.362 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.380+0000 7fb5c9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:14.362 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.380+0000 7fb5c9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:14.362 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.380+0000 7fb5c9ffb640 10 monclient: dump: 2024-03-29T11:19:14.362 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:19:14.363 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:19:14.363 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:14.363 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:14.363 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:14.363 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:19:14.363 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:19:14.363 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:19:14.363 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:19:14.363 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:19:14.363 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.380+0000 7fb5c9ffb640 1 -- 172.21.15.47:0/4058192102 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7fb5d001a960 con 0x7fb5cc038c60 2024-03-29T11:19:14.363 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.381+0000 7fb5c9ffb640 1 -- 172.21.15.47:0/4058192102 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7fb5d0084890 con 0x7fb5cc038c60 2024-03-29T11:19:14.363 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.381+0000 7fb5c9ffb640 10 client.4487.objecter ms_dispatch 0x7fb5d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:14.363 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.381+0000 7fb5c9ffb640 3 client.4487.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:14.363 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.381+0000 7fb5c9ffb640 3 client.4487.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:14.364 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.381+0000 7fb5c9ffb640 20 client.4487.objecter dump_active .. 0 homeless 2024-03-29T11:19:14.364 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.381+0000 7fb5d8a21640 10 monclient: get_auth_request con 0x7fb5b806e150 auth_method 0 2024-03-29T11:19:14.364 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.381+0000 7fb5c9ffb640 10 client.4487.objecter ms_handle_connect 0x7fb5b806e150 2024-03-29T11:19:14.364 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.383+0000 7fb5da224640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:14.364 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.383+0000 7fb5da224640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:14.364 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.383+0000 7fb5da224640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:19:14.364 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.383+0000 7fb5da224640 1 -- 172.21.15.47:0/4058192102 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5b4005080 con 0x7fb5cc038c60 2024-03-29T11:19:14.368 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.391+0000 7fb5c9ffb640 1 -- 172.21.15.47:0/4058192102 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb5d00501f0 con 0x7fb5cc038c60 2024-03-29T11:19:14.368 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.391+0000 7fb5c9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:14.368 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.391+0000 7fb5c9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:14.707 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.730+0000 7fb5da224640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:14.707 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.730+0000 7fb5da224640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:14.707 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.730+0000 7fb5da224640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:19:14.707 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.730+0000 7fb5da224640 1 -- 172.21.15.47:0/4058192102 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb5b4005330 con 0x7fb5cc038c60 2024-03-29T11:19:14.711 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.732+0000 7fb5c9ffb640 1 -- 172.21.15.47:0/4058192102 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7fb5d0052120 con 0x7fb5cc038c60 2024-03-29T11:19:14.711 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.732+0000 7fb5c9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:14.711 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.732+0000 7fb5c9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:14.711 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:19:14.711 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:14.717 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.738+0000 7fb5da224640 10 librados: watch_flush enter 2024-03-29T11:19:14.717 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.738+0000 7fb5da224640 10 librados: watch_flush exit 2024-03-29T11:19:14.717 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.738+0000 7fb5da224640 20 client.4487.objecter shutdown clearing up homeless session... 2024-03-29T11:19:14.718 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.738+0000 7fb5da224640 10 client.4487.objecter successfully canceled tick 2024-03-29T11:19:14.718 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.741+0000 7fb5da224640 1 -- 172.21.15.47:0/4058192102 >> v1:172.21.15.121:6816/1964117370 conn(0x7fb5b806e150 legacy=0x7fb5b8070610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:14.718 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.741+0000 7fb5da224640 10 monclient: shutdown 2024-03-29T11:19:14.718 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.741+0000 7fb5da224640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:14.719 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.741+0000 7fb5da224640 1 -- 172.21.15.47:0/4058192102 >> v1:172.21.15.47:6790/0 conn(0x7fb5cc038c60 legacy=0x7fb5cc15ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:14.719 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.741+0000 7fb5da224640 1 -- 172.21.15.47:0/4058192102 shutdown_connections 2024-03-29T11:19:14.719 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.741+0000 7fb5da224640 1 -- 172.21.15.47:0/4058192102 >> 172.21.15.47:0/4058192102 conn(0x7fb5cc005c20 msgr2=0x7fb5cc03bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:14.719 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.742+0000 7fb5da224640 1 -- 172.21.15.47:0/4058192102 shutdown_connections 2024-03-29T11:19:14.719 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.742+0000 7fb5da224640 1 -- 172.21.15.47:0/4058192102 wait complete. 2024-03-29T11:19:14.719 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:14.742+0000 7fb5da224640 1 librados: shutdown 2024-03-29T11:19:14.739 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:21.546 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:21.695 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.717+0000 7f8d416c8640 10 monclient: get_monmap_and_config 2024-03-29T11:19:21.696 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.717+0000 7f8d416c8640 10 monclient: build_initial_monmap 2024-03-29T11:19:21.696 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.717+0000 7f8d416c8640 10 monclient: monmap: 2024-03-29T11:19:21.696 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:19:21.696 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:19:21.696 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:19:21.718741+0000 2024-03-29T11:19:21.696 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:19:21.718741+0000 2024-03-29T11:19:21.696 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:21.696 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:19:21.696 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:19:21.696 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:19:21.696 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:19:21.696 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:19:21.699 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.720+0000 7f8d416c8640 1 Processor -- start 2024-03-29T11:19:21.699 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.720+0000 7f8d416c8640 1 -- start start 2024-03-29T11:19:21.699 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.720+0000 7f8d416c8640 10 monclient: init 2024-03-29T11:19:21.699 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.721+0000 7f8d416c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:21.699 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.721+0000 7f8d416c8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:19:21.699 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.721+0000 7f8d416c8640 10 monclient(hunting): picked mon.noname-a con 0x7f8d3c0a9930 addr v1:172.21.15.47:6789/0 2024-03-29T11:19:21.699 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.721+0000 7f8d416c8640 10 monclient(hunting): picked mon.noname-c con 0x7f8d3c149dc0 addr v1:172.21.15.121:6789/0 2024-03-29T11:19:21.699 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.721+0000 7f8d416c8640 10 monclient(hunting): picked mon.noname-b con 0x7f8d3c14ff40 addr v1:172.21.15.47:6790/0 2024-03-29T11:19:21.699 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.721+0000 7f8d416c8640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d3c0aae60 con 0x7f8d3c0a9930 2024-03-29T11:19:21.699 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.721+0000 7f8d416c8640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d3c0b4310 con 0x7f8d3c149dc0 2024-03-29T11:19:21.699 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.721+0000 7f8d416c8640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d3c0afdd0 con 0x7f8d3c14ff40 2024-03-29T11:19:21.699 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.721+0000 7f8d416c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:21.699 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.721+0000 7f8d416c8640 10 monclient(hunting): authenticate will time out at 1204.909477s 2024-03-29T11:19:21.699 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.722+0000 7f8d3b7fe640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f8d3c14ff40 0x7f8d3c152330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:33596/0 (socket says 172.21.15.47:33596) 2024-03-29T11:19:21.699 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.722+0000 7f8d3b7fe640 1 -- 172.21.15.47:0/1906518293 learned_addr learned my addr 172.21.15.47:0/1906518293 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:19:21.702 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.724+0000 7f8d39ffb640 1 -- 172.21.15.47:0/1906518293 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 333496219 0 0) 0x7f8d3c0afdd0 con 0x7f8d3c14ff40 2024-03-29T11:19:21.702 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.724+0000 7f8d39ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:21.702 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.724+0000 7f8d39ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:21.702 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.724+0000 7f8d39ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:21.702 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.724+0000 7f8d39ffb640 10 monclient(hunting): my global_id is 4490 2024-03-29T11:19:21.702 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.724+0000 7f8d39ffb640 1 -- 172.21.15.47:0/1906518293 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d24003520 con 0x7f8d3c14ff40 2024-03-29T11:19:21.702 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.725+0000 7f8d39ffb640 1 -- 172.21.15.47:0/1906518293 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1238741444 0 0) 0x7f8d3c0b4310 con 0x7f8d3c149dc0 2024-03-29T11:19:21.702 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.725+0000 7f8d39ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:21.702 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.725+0000 7f8d39ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:21.703 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.725+0000 7f8d39ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:21.703 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.725+0000 7f8d39ffb640 10 monclient(hunting): my global_id is 4444 2024-03-29T11:19:21.703 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.725+0000 7f8d39ffb640 1 -- 172.21.15.47:0/1906518293 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d3c0afdd0 con 0x7f8d3c149dc0 2024-03-29T11:19:21.704 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.726+0000 7f8d39ffb640 1 -- 172.21.15.47:0/1906518293 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2097204249 0 0) 0x7f8d24003520 con 0x7f8d3c14ff40 2024-03-29T11:19:21.704 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.726+0000 7f8d39ffb640 1 -- 172.21.15.47:0/1906518293 >> v1:172.21.15.121:6789/0 conn(0x7f8d3c149dc0 legacy=0x7f8d3c14a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:21.705 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.726+0000 7f8d39ffb640 1 -- 172.21.15.47:0/1906518293 >> v1:172.21.15.47:6789/0 conn(0x7f8d3c0a9930 legacy=0x7f8d3c143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:21.705 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.726+0000 7f8d39ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:19:21.705 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.726+0000 7f8d39ffb640 1 monclient: found mon.noname-b 2024-03-29T11:19:21.705 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.726+0000 7f8d39ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:21.705 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.726+0000 7f8d39ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-29T11:19:21.705 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.726+0000 7f8d39ffb640 1 -- 172.21.15.47:0/1906518293 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d3c14a950 con 0x7f8d3c14ff40 2024-03-29T11:19:21.705 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.726+0000 7f8d39ffb640 10 monclient: _finish_auth 0 2024-03-29T11:19:21.705 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.726+0000 7f8d39ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:19:21.705 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.726+0000 7f8d39ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:21.705 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.726+0000 7f8d39ffb640 1 -- 172.21.15.47:0/1906518293 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f8d34001a10 con 0x7f8d3c14ff40 2024-03-29T11:19:21.705 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.726+0000 7f8d416c8640 5 monclient: authenticate success, global_id 4490 2024-03-29T11:19:21.705 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.726+0000 7f8d416c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:21.705 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.726+0000 7f8d39ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:21.705 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.726+0000 7f8d39ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:21.705 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.726+0000 7f8d39ffb640 10 monclient: dump: 2024-03-29T11:19:21.706 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:19:21.706 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:19:21.706 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:21.706 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:21.706 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:21.706 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:19:21.706 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:19:21.706 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:19:21.706 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:19:21.706 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:19:21.706 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.726+0000 7f8d416c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:21.708 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.729+0000 7f8d39ffb640 1 -- 172.21.15.47:0/1906518293 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d34001c30 con 0x7f8d3c14ff40 2024-03-29T11:19:21.708 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.729+0000 7f8d39ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:21.708 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.729+0000 7f8d39ffb640 1 -- 172.21.15.47:0/1906518293 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f8d34001fc0 con 0x7f8d3c14ff40 2024-03-29T11:19:21.708 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.729+0000 7f8d39ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:21.708 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.729+0000 7f8d39ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:21.708 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.729+0000 7f8d39ffb640 10 monclient: dump: 2024-03-29T11:19:21.708 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:19:21.708 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:19:21.708 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:21.708 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:21.708 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:21.708 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:19:21.708 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:19:21.708 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:19:21.709 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:19:21.709 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:19:21.709 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.729+0000 7f8d416c8640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:21.709 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.729+0000 7f8d416c8640 4 set_mon_vals no callback set 2024-03-29T11:19:21.709 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.730+0000 7f8d416c8640 10 monclient: shutdown 2024-03-29T11:19:21.709 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.730+0000 7f8d416c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:21.709 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.730+0000 7f8d416c8640 1 -- 172.21.15.47:0/1906518293 >> v1:172.21.15.47:6790/0 conn(0x7f8d3c14ff40 legacy=0x7f8d3c152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:21.709 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.731+0000 7f8d416c8640 1 -- 172.21.15.47:0/1906518293 shutdown_connections 2024-03-29T11:19:21.709 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.731+0000 7f8d416c8640 1 -- 172.21.15.47:0/1906518293 >> 172.21.15.47:0/1906518293 conn(0x7f8d3c0b35e0 msgr2=0x7f8d3c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:21.709 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.732+0000 7f8d416c8640 1 -- 172.21.15.47:0/1906518293 shutdown_connections 2024-03-29T11:19:21.710 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.732+0000 7f8d416c8640 1 -- 172.21.15.47:0/1906518293 wait complete. 2024-03-29T11:19:21.710 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.733+0000 7f8d416c8640 10 monclient: build_initial_monmap 2024-03-29T11:19:21.710 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.733+0000 7f8d416c8640 10 monclient: monmap: 2024-03-29T11:19:21.710 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:19:21.710 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:21.710 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:19:21.710 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:19:21.710 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:21.710 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:19:21.710 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:19:21.711 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:19:21.711 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:19:21.711 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:19:21.712 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.734+0000 7f8d416c8640 1 librados: starting msgr at 2024-03-29T11:19:21.712 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.734+0000 7f8d416c8640 1 librados: starting objecter 2024-03-29T11:19:21.712 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.734+0000 7f8d416c8640 1 Processor -- start 2024-03-29T11:19:21.712 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.734+0000 7f8d416c8640 1 -- start start 2024-03-29T11:19:21.712 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.734+0000 7f8d416c8640 1 librados: setting wanted keys 2024-03-29T11:19:21.712 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.734+0000 7f8d416c8640 1 librados: calling monclient init 2024-03-29T11:19:21.712 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.734+0000 7f8d416c8640 10 monclient: init 2024-03-29T11:19:21.713 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.736+0000 7f8d416c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:21.713 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.736+0000 7f8d416c8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:19:21.715 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.738+0000 7f8d416c8640 10 monclient(hunting): picked mon.noname-b con 0x7f8d3c0a9930 addr v1:172.21.15.121:6789/0 2024-03-29T11:19:21.715 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.738+0000 7f8d416c8640 10 monclient(hunting): picked mon.noname-c con 0x7f8d3c149dc0 addr v1:172.21.15.47:6790/0 2024-03-29T11:19:21.725 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.738+0000 7f8d3a7fc640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f8d3c149dc0 0x7f8d3c1dbf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:36838/0 (socket says 172.21.15.47:36838) 2024-03-29T11:19:21.725 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.738+0000 7f8d3affd640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f8d3c0a9930 0x7f8d3c0abf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.47:49607/0 (socket says 172.21.15.47:49607) 2024-03-29T11:19:21.725 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.739+0000 7f8d416c8640 10 monclient(hunting): picked mon.noname-a con 0x7f8d3c14ff40 addr v1:172.21.15.47:6789/0 2024-03-29T11:19:21.725 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.739+0000 7f8d416c8640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d3c1dbad0 con 0x7f8d3c14ff40 2024-03-29T11:19:21.725 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.739+0000 7f8d416c8640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d3c148090 con 0x7f8d3c0a9930 2024-03-29T11:19:21.725 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.739+0000 7f8d416c8640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d3c1db6d0 con 0x7f8d3c149dc0 2024-03-29T11:19:21.725 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.739+0000 7f8d416c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:21.725 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.739+0000 7f8d416c8640 10 monclient(hunting): authenticate will time out at 1204.927099s 2024-03-29T11:19:21.725 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.739+0000 7f8d3a7fc640 1 -- 172.21.15.47:0/1163461796 learned_addr learned my addr 172.21.15.47:0/1163461796 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:19:21.725 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.742+0000 7f8d237fe640 10 client.?.objecter ms_handle_connect 0x7f8d3c149dc0 2024-03-29T11:19:21.726 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.742+0000 7f8d237fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:21.726 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.742+0000 7f8d237fe640 1 -- 172.21.15.47:0/1163461796 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3967441950 0 0) 0x7f8d3c1db6d0 con 0x7f8d3c149dc0 2024-03-29T11:19:21.726 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.742+0000 7f8d237fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:21.726 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.742+0000 7f8d237fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:21.726 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.742+0000 7f8d237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:21.726 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.742+0000 7f8d237fe640 10 monclient(hunting): my global_id is 4493 2024-03-29T11:19:21.726 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.742+0000 7f8d237fe640 1 -- 172.21.15.47:0/1163461796 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d28004eb0 con 0x7f8d3c149dc0 2024-03-29T11:19:21.726 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.742+0000 7f8d237fe640 1 -- 172.21.15.47:0/1163461796 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3031896380 0 0) 0x7f8d28004eb0 con 0x7f8d3c149dc0 2024-03-29T11:19:21.726 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.742+0000 7f8d237fe640 1 -- 172.21.15.47:0/1163461796 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d3c1db6d0 con 0x7f8d3c149dc0 2024-03-29T11:19:21.726 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.742+0000 7f8d237fe640 1 -- 172.21.15.47:0/1163461796 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f8d30001a50 con 0x7f8d3c149dc0 2024-03-29T11:19:21.726 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.742+0000 7f8d237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:21.726 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.742+0000 7f8d237fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:21.727 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.742+0000 7f8d237fe640 10 monclient(hunting): dump: 2024-03-29T11:19:21.727 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:19:21.727 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:19:21.727 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:21.727 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:21.727 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:21.727 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:19:21.727 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:19:21.727 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:19:21.727 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:19:21.727 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:19:21.727 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.742+0000 7f8d237fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:21.727 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.743+0000 7f8d237fe640 1 -- 172.21.15.47:0/1163461796 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1659562180 0 0) 0x7f8d3c1db6d0 con 0x7f8d3c149dc0 2024-03-29T11:19:21.727 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.743+0000 7f8d237fe640 1 -- 172.21.15.47:0/1163461796 >> v1:172.21.15.121:6789/0 conn(0x7f8d3c0a9930 legacy=0x7f8d3c0abf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:21.728 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.743+0000 7f8d237fe640 1 -- 172.21.15.47:0/1163461796 >> v1:172.21.15.47:6789/0 conn(0x7f8d3c14ff40 legacy=0x7f8d3c148b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:21.728 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.743+0000 7f8d237fe640 10 monclient: _finish_hunting 0 2024-03-29T11:19:21.728 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.743+0000 7f8d237fe640 1 monclient: found mon.c 2024-03-29T11:19:21.728 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.743+0000 7f8d237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:21.728 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.743+0000 7f8d237fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:19:21.728 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.743+0000 7f8d237fe640 1 -- 172.21.15.47:0/1163461796 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d3c149210 con 0x7f8d3c149dc0 2024-03-29T11:19:21.728 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.743+0000 7f8d237fe640 10 monclient: _finish_auth 0 2024-03-29T11:19:21.728 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.743+0000 7f8d237fe640 10 monclient: _check_auth_tickets 2024-03-29T11:19:21.728 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.743+0000 7f8d237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:21.728 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.743+0000 7f8d416c8640 5 monclient: authenticate success, global_id 4493 2024-03-29T11:19:21.728 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.743+0000 7f8d416c8640 10 monclient: _renew_subs 2024-03-29T11:19:21.728 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.743+0000 7f8d416c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:19:21.728 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.743+0000 7f8d416c8640 1 -- 172.21.15.47:0/1163461796 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d3c149400 con 0x7f8d3c149dc0 2024-03-29T11:19:21.728 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.743+0000 7f8d237fe640 1 -- 172.21.15.47:0/1163461796 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d30001d30 con 0x7f8d3c149dc0 2024-03-29T11:19:21.728 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.743+0000 7f8d237fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:21.729 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.743+0000 7f8d237fe640 1 -- 172.21.15.47:0/1163461796 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f8d30001a50 con 0x7f8d3c149dc0 2024-03-29T11:19:21.729 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.743+0000 7f8d237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:21.729 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.743+0000 7f8d237fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:21.729 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.743+0000 7f8d237fe640 10 monclient: dump: 2024-03-29T11:19:21.729 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:19:21.729 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:19:21.729 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:21.729 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:21.729 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:21.729 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:19:21.729 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:19:21.729 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:19:21.729 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:19:21.729 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:19:21.729 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.743+0000 7f8d416c8640 10 client.4493.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:21.729 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.743+0000 7f8d416c8640 10 monclient: _renew_subs 2024-03-29T11:19:21.730 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.743+0000 7f8d416c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:19:21.730 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.743+0000 7f8d416c8640 1 -- 172.21.15.47:0/1163461796 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d3c0acaa0 con 0x7f8d3c149dc0 2024-03-29T11:19:21.730 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.743+0000 7f8d416c8640 1 librados: init done 2024-03-29T11:19:21.730 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.744+0000 7f8d397fa640 4 set_mon_vals no callback set 2024-03-29T11:19:21.730 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.745+0000 7f8d237fe640 1 -- 172.21.15.47:0/1163461796 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7f8d300039a0 con 0x7f8d3c149dc0 2024-03-29T11:19:21.730 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.745+0000 7f8d237fe640 1 -- 172.21.15.47:0/1163461796 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7f8d30084730 con 0x7f8d3c149dc0 2024-03-29T11:19:21.730 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.746+0000 7f8d237fe640 10 client.4493.objecter ms_dispatch 0x7f8d3c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:21.730 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.746+0000 7f8d237fe640 3 client.4493.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:21.730 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.746+0000 7f8d237fe640 3 client.4493.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:21.730 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.746+0000 7f8d237fe640 20 client.4493.objecter dump_active .. 0 homeless 2024-03-29T11:19:21.730 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.746+0000 7f8d3affd640 10 monclient: get_auth_request con 0x7f8d2806df10 auth_method 0 2024-03-29T11:19:21.730 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.746+0000 7f8d237fe640 10 client.4493.objecter ms_handle_connect 0x7f8d2806df10 2024-03-29T11:19:21.731 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.749+0000 7f8d416c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:21.731 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.749+0000 7f8d416c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:21.731 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.749+0000 7f8d416c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:19:21.731 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.749+0000 7f8d416c8640 1 -- 172.21.15.47:0/1163461796 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d1c005080 con 0x7f8d3c149dc0 2024-03-29T11:19:21.738 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.760+0000 7f8d237fe640 1 -- 172.21.15.47:0/1163461796 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8d3004fea0 con 0x7f8d3c149dc0 2024-03-29T11:19:21.738 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.760+0000 7f8d237fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:21.738 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:21.760+0000 7f8d237fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:22.073 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:22.095+0000 7f8d416c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:22.073 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:22.095+0000 7f8d416c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:22.073 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:22.095+0000 7f8d416c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:19:22.073 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:22.095+0000 7f8d416c8640 1 -- 172.21.15.47:0/1163461796 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8d1c005330 con 0x7f8d3c149dc0 2024-03-29T11:19:22.073 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:22.096+0000 7f8d237fe640 1 -- 172.21.15.47:0/1163461796 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7f8d300018b0 con 0x7f8d3c149dc0 2024-03-29T11:19:22.073 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:22.096+0000 7f8d237fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:22.074 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:22.096+0000 7f8d237fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:22.076 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:19:22.076 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:22.077 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:22.100+0000 7f8d416c8640 10 librados: watch_flush enter 2024-03-29T11:19:22.077 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:22.100+0000 7f8d416c8640 10 librados: watch_flush exit 2024-03-29T11:19:22.077 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:22.100+0000 7f8d416c8640 20 client.4493.objecter shutdown clearing up homeless session... 2024-03-29T11:19:22.077 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:22.100+0000 7f8d416c8640 10 client.4493.objecter successfully canceled tick 2024-03-29T11:19:22.077 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:22.100+0000 7f8d416c8640 1 -- 172.21.15.47:0/1163461796 >> v1:172.21.15.121:6816/1964117370 conn(0x7f8d2806df10 legacy=0x7f8d280703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:22.077 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:22.100+0000 7f8d416c8640 10 monclient: shutdown 2024-03-29T11:19:22.077 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:22.100+0000 7f8d416c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:22.078 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:22.100+0000 7f8d416c8640 1 -- 172.21.15.47:0/1163461796 >> v1:172.21.15.47:6790/0 conn(0x7f8d3c149dc0 legacy=0x7f8d3c1dbf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:22.078 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:22.100+0000 7f8d416c8640 1 -- 172.21.15.47:0/1163461796 shutdown_connections 2024-03-29T11:19:22.078 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:22.100+0000 7f8d416c8640 1 -- 172.21.15.47:0/1163461796 >> 172.21.15.47:0/1163461796 conn(0x7f8d3c0b35e0 msgr2=0x7f8d3c0b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:22.078 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:22.100+0000 7f8d416c8640 1 -- 172.21.15.47:0/1163461796 shutdown_connections 2024-03-29T11:19:22.078 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:22.100+0000 7f8d416c8640 1 -- 172.21.15.47:0/1163461796 wait complete. 2024-03-29T11:19:22.078 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:22.100+0000 7f8d416c8640 1 librados: shutdown 2024-03-29T11:19:22.094 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:28.901 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:29.049 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.072+0000 7f4064e05640 10 monclient: get_monmap_and_config 2024-03-29T11:19:29.049 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.072+0000 7f4064e05640 10 monclient: build_initial_monmap 2024-03-29T11:19:29.049 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.072+0000 7f4064e05640 10 monclient: monmap: 2024-03-29T11:19:29.049 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:19:29.049 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:19:29.049 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:19:29.073465+0000 2024-03-29T11:19:29.050 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:19:29.073465+0000 2024-03-29T11:19:29.050 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:29.050 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:19:29.050 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:19:29.050 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:19:29.050 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:19:29.050 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:19:29.051 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.074+0000 7f4064e05640 1 Processor -- start 2024-03-29T11:19:29.052 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.075+0000 7f4064e05640 1 -- start start 2024-03-29T11:19:29.052 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.075+0000 7f4064e05640 10 monclient: init 2024-03-29T11:19:29.053 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.076+0000 7f4064e05640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:29.055 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.076+0000 7f4064e05640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:29.055 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.076+0000 7f4064e05640 10 monclient(hunting): picked mon.noname-c con 0x7f40601457f0 addr v1:172.21.15.121:6789/0 2024-03-29T11:19:29.056 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.078+0000 7f4064e05640 10 monclient(hunting): picked mon.noname-a con 0x7f406013f650 addr v1:172.21.15.47:6789/0 2024-03-29T11:19:29.056 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.078+0000 7f4064e05640 10 monclient(hunting): picked mon.noname-b con 0x7f40600a9d70 addr v1:172.21.15.47:6790/0 2024-03-29T11:19:29.056 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.078+0000 7f4064e05640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40600aaef0 con 0x7f406013f650 2024-03-29T11:19:29.056 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.078+0000 7f4064e05640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4060145bd0 con 0x7f40601457f0 2024-03-29T11:19:29.056 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.078+0000 7f4064e05640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4060145eb0 con 0x7f40600a9d70 2024-03-29T11:19:29.056 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.078+0000 7f4064e05640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:29.056 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.078+0000 7f4064e05640 10 monclient(hunting): authenticate will time out at 1212.266140s 2024-03-29T11:19:29.056 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.078+0000 7f405dd9b640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f406013f650 0x7f40600b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:33568/0 (socket says 172.21.15.47:33568) 2024-03-29T11:19:29.056 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.078+0000 7f405dd9b640 1 -- 172.21.15.47:0/3087966464 learned_addr learned my addr 172.21.15.47:0/3087966464 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:19:29.057 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.080+0000 7f405d59a640 1 -- 172.21.15.47:0/3087966464 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1926386248 0 0) 0x7f40600aaef0 con 0x7f406013f650 2024-03-29T11:19:29.058 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.080+0000 7f405d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:19:29.058 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.080+0000 7f405d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:29.058 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.080+0000 7f405d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:29.058 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.080+0000 7f405d59a640 10 monclient(hunting): my global_id is 4440 2024-03-29T11:19:29.058 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.080+0000 7f405d59a640 1 -- 172.21.15.47:0/3087966464 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4048003520 con 0x7f406013f650 2024-03-29T11:19:29.058 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.080+0000 7f405d59a640 1 -- 172.21.15.47:0/3087966464 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 789586580 0 0) 0x7f4060145bd0 con 0x7f40601457f0 2024-03-29T11:19:29.058 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.080+0000 7f405d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:19:29.058 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.080+0000 7f405d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:29.058 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.080+0000 7f405d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:29.058 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.080+0000 7f405d59a640 10 monclient(hunting): my global_id is 4447 2024-03-29T11:19:29.058 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.080+0000 7f405d59a640 1 -- 172.21.15.47:0/3087966464 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40600aaef0 con 0x7f40601457f0 2024-03-29T11:19:29.060 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.082+0000 7f405d59a640 1 -- 172.21.15.47:0/3087966464 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3706990052 0 0) 0x7f40600aaef0 con 0x7f40601457f0 2024-03-29T11:19:29.060 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.082+0000 7f405d59a640 1 -- 172.21.15.47:0/3087966464 >> v1:172.21.15.47:6790/0 conn(0x7f40600a9d70 legacy=0x7f406014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:19:29.060 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.082+0000 7f405d59a640 1 -- 172.21.15.47:0/3087966464 >> v1:172.21.15.47:6789/0 conn(0x7f406013f650 legacy=0x7f40600b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:29.060 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.082+0000 7f405d59a640 10 monclient: _finish_hunting 0 2024-03-29T11:19:29.061 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.082+0000 7f405d59a640 1 monclient: found mon.noname-c 2024-03-29T11:19:29.061 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.083+0000 7f405d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:29.061 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.083+0000 7f405d59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:19:29.061 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.083+0000 7f405d59a640 1 -- 172.21.15.47:0/3087966464 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4060146340 con 0x7f40601457f0 2024-03-29T11:19:29.061 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.083+0000 7f405d59a640 10 monclient: _finish_auth 0 2024-03-29T11:19:29.061 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.083+0000 7f405d59a640 10 monclient: _check_auth_tickets 2024-03-29T11:19:29.061 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.083+0000 7f405d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:29.061 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.083+0000 7f4064e05640 5 monclient: authenticate success, global_id 4447 2024-03-29T11:19:29.061 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.083+0000 7f4064e05640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:29.063 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.083+0000 7f405d59a640 1 -- 172.21.15.47:0/3087966464 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f40580019b0 con 0x7f40601457f0 2024-03-29T11:19:29.063 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.084+0000 7f405d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:29.063 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.084+0000 7f405d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:29.063 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.084+0000 7f405d59a640 10 monclient: dump: 2024-03-29T11:19:29.063 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:19:29.063 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:19:29.063 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:29.063 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:29.063 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:29.063 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:19:29.064 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:19:29.064 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:19:29.064 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:19:29.064 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:19:29.064 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.084+0000 7f405d59a640 1 -- 172.21.15.47:0/3087966464 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4058001c00 con 0x7f40601457f0 2024-03-29T11:19:29.064 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.084+0000 7f405d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:29.064 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.084+0000 7f405d59a640 1 -- 172.21.15.47:0/3087966464 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f4058001f90 con 0x7f40601457f0 2024-03-29T11:19:29.064 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.084+0000 7f405d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:29.064 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.084+0000 7f405d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:29.064 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.084+0000 7f405d59a640 10 monclient: dump: 2024-03-29T11:19:29.064 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:19:29.064 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:19:29.064 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:29.064 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:29.065 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:29.065 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:19:29.065 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:19:29.065 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:19:29.065 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:19:29.065 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:19:29.065 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.084+0000 7f4064e05640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:29.065 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.084+0000 7f4064e05640 4 set_mon_vals no callback set 2024-03-29T11:19:29.065 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.085+0000 7f4064e05640 10 monclient: shutdown 2024-03-29T11:19:29.065 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.085+0000 7f4064e05640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:29.065 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.085+0000 7f4064e05640 1 -- 172.21.15.47:0/3087966464 >> v1:172.21.15.121:6789/0 conn(0x7f40601457f0 legacy=0x7f406013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:29.066 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.087+0000 7f4064e05640 1 -- 172.21.15.47:0/3087966464 shutdown_connections 2024-03-29T11:19:29.066 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.087+0000 7f4064e05640 1 -- 172.21.15.47:0/3087966464 >> 172.21.15.47:0/3087966464 conn(0x7f4060066a00 msgr2=0x7f4060066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:29.066 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.088+0000 7f4064e05640 1 -- 172.21.15.47:0/3087966464 shutdown_connections 2024-03-29T11:19:29.066 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.088+0000 7f4064e05640 1 -- 172.21.15.47:0/3087966464 wait complete. 2024-03-29T11:19:29.066 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.089+0000 7f4064e05640 10 monclient: build_initial_monmap 2024-03-29T11:19:29.066 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.089+0000 7f4064e05640 10 monclient: monmap: 2024-03-29T11:19:29.067 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:19:29.067 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:29.067 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:19:29.067 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:19:29.067 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:29.067 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:19:29.067 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:19:29.067 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:19:29.067 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:19:29.067 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:19:29.067 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.089+0000 7f4064e05640 1 librados: starting msgr at 2024-03-29T11:19:29.067 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.089+0000 7f4064e05640 1 librados: starting objecter 2024-03-29T11:19:29.067 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.089+0000 7f4064e05640 1 Processor -- start 2024-03-29T11:19:29.068 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.089+0000 7f4064e05640 1 -- start start 2024-03-29T11:19:29.068 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.089+0000 7f4064e05640 1 librados: setting wanted keys 2024-03-29T11:19:29.068 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.089+0000 7f4064e05640 1 librados: calling monclient init 2024-03-29T11:19:29.068 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.089+0000 7f4064e05640 10 monclient: init 2024-03-29T11:19:29.068 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.090+0000 7f4064e05640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:29.068 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.090+0000 7f4064e05640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:29.068 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.090+0000 7f4064e05640 10 monclient(hunting): picked mon.noname-c con 0x7f40600a9d70 addr v1:172.21.15.47:6790/0 2024-03-29T11:19:29.068 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.090+0000 7f4064e05640 10 monclient(hunting): picked mon.noname-b con 0x7f406013f650 addr v1:172.21.15.121:6789/0 2024-03-29T11:19:29.068 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.090+0000 7f4064e05640 10 monclient(hunting): picked mon.noname-a con 0x7f40601457f0 addr v1:172.21.15.47:6789/0 2024-03-29T11:19:29.069 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.090+0000 7f4064e05640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f406013a8f0 con 0x7f40601457f0 2024-03-29T11:19:29.069 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.090+0000 7f4064e05640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4060139cd0 con 0x7f406013f650 2024-03-29T11:19:29.069 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.090+0000 7f4064e05640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f406013ab60 con 0x7f40600a9d70 2024-03-29T11:19:29.069 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.090+0000 7f4064e05640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:29.069 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.090+0000 7f4064e05640 10 monclient(hunting): authenticate will time out at 1212.278866s 2024-03-29T11:19:29.070 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.092+0000 7f405ed9d640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f40601457f0 0x7f40601e66d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:36654/0 (socket says 172.21.15.47:36654) 2024-03-29T11:19:29.070 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.092+0000 7f405ed9d640 1 -- 172.21.15.47:0/3698887903 learned_addr learned my addr 172.21.15.47:0/3698887903 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:19:29.070 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.092+0000 7f404effd640 10 client.?.objecter ms_handle_connect 0x7f40601457f0 2024-03-29T11:19:29.070 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.092+0000 7f404effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:29.070 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.092+0000 7f404effd640 1 -- 172.21.15.47:0/3698887903 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1979928151 0 0) 0x7f406013a8f0 con 0x7f40601457f0 2024-03-29T11:19:29.070 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.092+0000 7f404effd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:29.070 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.092+0000 7f404effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:29.070 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.092+0000 7f404effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:29.070 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.092+0000 7f404effd640 10 monclient(hunting): my global_id is 4443 2024-03-29T11:19:29.070 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.092+0000 7f404effd640 1 -- 172.21.15.47:0/3698887903 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4040004eb0 con 0x7f40601457f0 2024-03-29T11:19:29.070 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.093+0000 7f404effd640 1 -- 172.21.15.47:0/3698887903 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3943289350 0 0) 0x7f4040004eb0 con 0x7f40601457f0 2024-03-29T11:19:29.070 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.093+0000 7f404effd640 1 -- 172.21.15.47:0/3698887903 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f406013a8f0 con 0x7f40601457f0 2024-03-29T11:19:29.071 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.093+0000 7f404effd640 1 -- 172.21.15.47:0/3698887903 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f4050001a60 con 0x7f40601457f0 2024-03-29T11:19:29.071 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.093+0000 7f404effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:29.071 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.093+0000 7f404effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:29.071 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.093+0000 7f404effd640 10 monclient(hunting): dump: 2024-03-29T11:19:29.071 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:19:29.071 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:19:29.071 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:29.071 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:29.071 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:29.071 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:19:29.071 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:19:29.071 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:19:29.071 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:19:29.071 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:19:29.072 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.093+0000 7f404effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:29.072 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.093+0000 7f404effd640 1 -- 172.21.15.47:0/3698887903 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1088932807 0 0) 0x7f406013a8f0 con 0x7f40601457f0 2024-03-29T11:19:29.072 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.093+0000 7f404effd640 1 -- 172.21.15.47:0/3698887903 >> v1:172.21.15.47:6790/0 conn(0x7f40600a9d70 legacy=0x7f40601403d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:19:29.073 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.093+0000 7f404effd640 1 -- 172.21.15.47:0/3698887903 >> v1:172.21.15.121:6789/0 conn(0x7f406013f650 legacy=0x7f4060140ae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:19:29.073 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.093+0000 7f404effd640 10 monclient: _finish_hunting 0 2024-03-29T11:19:29.073 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.093+0000 7f404effd640 1 monclient: found mon.a 2024-03-29T11:19:29.073 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.093+0000 7f404effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:29.073 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.094+0000 7f404effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:19:29.073 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.094+0000 7f404effd640 1 -- 172.21.15.47:0/3698887903 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f406013ae40 con 0x7f40601457f0 2024-03-29T11:19:29.073 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.094+0000 7f404effd640 10 monclient: _finish_auth 0 2024-03-29T11:19:29.074 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.094+0000 7f404effd640 10 monclient: _check_auth_tickets 2024-03-29T11:19:29.074 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.094+0000 7f404effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:29.074 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.094+0000 7f4064e05640 5 monclient: authenticate success, global_id 4443 2024-03-29T11:19:29.074 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.094+0000 7f4064e05640 10 monclient: _renew_subs 2024-03-29T11:19:29.074 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.094+0000 7f4064e05640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:19:29.074 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.094+0000 7f4064e05640 1 -- 172.21.15.47:0/3698887903 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40601411f0 con 0x7f40601457f0 2024-03-29T11:19:29.074 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.094+0000 7f4064e05640 10 client.4443.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:29.074 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.094+0000 7f4064e05640 10 monclient: _renew_subs 2024-03-29T11:19:29.074 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.094+0000 7f4064e05640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:19:29.074 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.094+0000 7f4064e05640 1 -- 172.21.15.47:0/3698887903 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4060141730 con 0x7f40601457f0 2024-03-29T11:19:29.074 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.094+0000 7f4064e05640 1 librados: init done 2024-03-29T11:19:29.074 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.094+0000 7f404effd640 1 -- 172.21.15.47:0/3698887903 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4050001bc0 con 0x7f40601457f0 2024-03-29T11:19:29.074 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.094+0000 7f404effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:29.074 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.094+0000 7f404effd640 1 -- 172.21.15.47:0/3698887903 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f4050001740 con 0x7f40601457f0 2024-03-29T11:19:29.075 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.094+0000 7f404ffff640 4 set_mon_vals no callback set 2024-03-29T11:19:29.075 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.094+0000 7f404effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:29.075 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.094+0000 7f404effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:29.075 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.094+0000 7f404effd640 10 monclient: dump: 2024-03-29T11:19:29.075 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:19:29.075 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:19:29.075 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:29.075 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:29.075 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:29.075 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:19:29.075 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:19:29.075 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:19:29.075 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:19:29.075 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:19:29.075 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.096+0000 7f4064e05640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:29.075 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.096+0000 7f4064e05640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:29.076 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.096+0000 7f4064e05640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:19:29.077 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.096+0000 7f4064e05640 1 -- 172.21.15.47:0/3698887903 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4060075f10 con 0x7f40601457f0 2024-03-29T11:19:29.077 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.099+0000 7f404effd640 1 -- 172.21.15.47:0/3698887903 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7f4050003b40 con 0x7f40601457f0 2024-03-29T11:19:29.077 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.099+0000 7f404effd640 1 -- 172.21.15.47:0/3698887903 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7f4050084b40 con 0x7f40601457f0 2024-03-29T11:19:29.077 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.099+0000 7f404effd640 10 client.4443.objecter ms_dispatch 0x7f4060000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:29.077 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.099+0000 7f404effd640 3 client.4443.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:29.077 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.100+0000 7f404effd640 3 client.4443.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:29.077 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.100+0000 7f404effd640 20 client.4443.objecter dump_active .. 0 homeless 2024-03-29T11:19:29.077 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.100+0000 7f405e59c640 10 monclient: get_auth_request con 0x7f404006db30 auth_method 0 2024-03-29T11:19:29.078 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.101+0000 7f404effd640 10 client.4443.objecter ms_handle_connect 0x7f404006db30 2024-03-29T11:19:29.083 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.106+0000 7f404effd640 1 -- 172.21.15.47:0/3698887903 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4050050420 con 0x7f40601457f0 2024-03-29T11:19:29.083 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.106+0000 7f404effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:29.083 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.106+0000 7f404effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:29.401 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.424+0000 7f4064e05640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:29.401 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.424+0000 7f4064e05640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:29.401 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.424+0000 7f4064e05640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:19:29.401 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.424+0000 7f4064e05640 1 -- 172.21.15.47:0/3698887903 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4060141380 con 0x7f40601457f0 2024-03-29T11:19:29.403 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.426+0000 7f404effd640 1 -- 172.21.15.47:0/3698887903 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7f4050052350 con 0x7f40601457f0 2024-03-29T11:19:29.403 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.426+0000 7f404effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:29.404 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.426+0000 7f404effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:29.404 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:19:29.404 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:29.407 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.430+0000 7f4064e05640 10 librados: watch_flush enter 2024-03-29T11:19:29.407 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.430+0000 7f4064e05640 10 librados: watch_flush exit 2024-03-29T11:19:29.407 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.430+0000 7f4064e05640 20 client.4443.objecter shutdown clearing up homeless session... 2024-03-29T11:19:29.407 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.430+0000 7f4064e05640 10 client.4443.objecter successfully canceled tick 2024-03-29T11:19:29.407 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.430+0000 7f4064e05640 1 -- 172.21.15.47:0/3698887903 >> v1:172.21.15.121:6816/1964117370 conn(0x7f404006db30 legacy=0x7f404006fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:29.407 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.430+0000 7f4064e05640 10 monclient: shutdown 2024-03-29T11:19:29.407 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.430+0000 7f4064e05640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:29.408 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.430+0000 7f4064e05640 1 -- 172.21.15.47:0/3698887903 >> v1:172.21.15.47:6789/0 conn(0x7f40601457f0 legacy=0x7f40601e66d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:29.408 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.430+0000 7f4064e05640 1 -- 172.21.15.47:0/3698887903 shutdown_connections 2024-03-29T11:19:29.408 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.430+0000 7f4064e05640 1 -- 172.21.15.47:0/3698887903 >> 172.21.15.47:0/3698887903 conn(0x7f4060066a00 msgr2=0x7f40600b4800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:29.410 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.433+0000 7f4064e05640 1 -- 172.21.15.47:0/3698887903 shutdown_connections 2024-03-29T11:19:29.410 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.434+0000 7f4064e05640 1 -- 172.21.15.47:0/3698887903 wait complete. 2024-03-29T11:19:29.411 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:29.434+0000 7f4064e05640 1 librados: shutdown 2024-03-29T11:19:29.427 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:36.234 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:36.386 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.409+0000 7fb2d12d3640 10 monclient: get_monmap_and_config 2024-03-29T11:19:36.387 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.409+0000 7fb2d12d3640 10 monclient: build_initial_monmap 2024-03-29T11:19:36.387 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.409+0000 7fb2d12d3640 10 monclient: monmap: 2024-03-29T11:19:36.387 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:19:36.387 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:19:36.387 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:19:36.410070+0000 2024-03-29T11:19:36.388 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:19:36.410070+0000 2024-03-29T11:19:36.388 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:36.388 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:19:36.388 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:19:36.388 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:19:36.389 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:19:36.389 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:19:36.391 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.411+0000 7fb2d12d3640 1 Processor -- start 2024-03-29T11:19:36.391 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.411+0000 7fb2d12d3640 1 -- start start 2024-03-29T11:19:36.391 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.411+0000 7fb2d12d3640 10 monclient: init 2024-03-29T11:19:36.391 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.412+0000 7fb2d12d3640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:36.391 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.412+0000 7fb2d12d3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:36.391 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.412+0000 7fb2d12d3640 10 monclient(hunting): picked mon.noname-c con 0x7fb2cc0a9930 addr v1:172.21.15.121:6789/0 2024-03-29T11:19:36.392 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.412+0000 7fb2d12d3640 10 monclient(hunting): picked mon.noname-a con 0x7fb2cc149dc0 addr v1:172.21.15.47:6789/0 2024-03-29T11:19:36.392 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.412+0000 7fb2d12d3640 10 monclient(hunting): picked mon.noname-b con 0x7fb2cc14ff40 addr v1:172.21.15.47:6790/0 2024-03-29T11:19:36.392 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.412+0000 7fb2d12d3640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2cc0aae60 con 0x7fb2cc149dc0 2024-03-29T11:19:36.392 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.412+0000 7fb2d12d3640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2cc0b4310 con 0x7fb2cc0a9930 2024-03-29T11:19:36.392 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.412+0000 7fb2d12d3640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2cc0afdd0 con 0x7fb2cc14ff40 2024-03-29T11:19:36.392 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.412+0000 7fb2d12d3640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:36.392 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.412+0000 7fb2d12d3640 10 monclient(hunting): authenticate will time out at 1219.600794s 2024-03-29T11:19:36.393 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.417+0000 7fb2caffd640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fb2cc0a9930 0x7fb2cc143a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.47:51569/0 (socket says 172.21.15.47:51569) 2024-03-29T11:19:36.393 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.417+0000 7fb2caffd640 1 -- 172.21.15.47:0/2922669192 learned_addr learned my addr 172.21.15.47:0/2922669192 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:19:36.394 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.417+0000 7fb2c9ffb640 1 -- 172.21.15.47:0/2922669192 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1500481715 0 0) 0x7fb2cc0b4310 con 0x7fb2cc0a9930 2024-03-29T11:19:36.394 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.417+0000 7fb2c9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:36.394 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.417+0000 7fb2c9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:36.394 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.417+0000 7fb2c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:36.394 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.417+0000 7fb2c9ffb640 10 monclient(hunting): my global_id is 4450 2024-03-29T11:19:36.394 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.417+0000 7fb2c9ffb640 1 -- 172.21.15.47:0/2922669192 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2b4003520 con 0x7fb2cc0a9930 2024-03-29T11:19:36.396 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.419+0000 7fb2c9ffb640 1 -- 172.21.15.47:0/2922669192 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4067093347 0 0) 0x7fb2cc0afdd0 con 0x7fb2cc14ff40 2024-03-29T11:19:36.396 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.419+0000 7fb2c9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:36.396 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.419+0000 7fb2c9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:36.396 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.419+0000 7fb2c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:36.396 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.419+0000 7fb2c9ffb640 10 monclient(hunting): my global_id is 4496 2024-03-29T11:19:36.397 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.419+0000 7fb2c9ffb640 1 -- 172.21.15.47:0/2922669192 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2cc0b4310 con 0x7fb2cc14ff40 2024-03-29T11:19:36.399 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.422+0000 7fb2c9ffb640 1 -- 172.21.15.47:0/2922669192 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3089101877 0 0) 0x7fb2b4003520 con 0x7fb2cc0a9930 2024-03-29T11:19:36.399 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.422+0000 7fb2c9ffb640 1 -- 172.21.15.47:0/2922669192 >> v1:172.21.15.47:6790/0 conn(0x7fb2cc14ff40 legacy=0x7fb2cc152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:36.399 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.422+0000 7fb2c9ffb640 1 -- 172.21.15.47:0/2922669192 >> v1:172.21.15.47:6789/0 conn(0x7fb2cc149dc0 legacy=0x7fb2cc14a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:36.399 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.422+0000 7fb2c9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:19:36.400 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.422+0000 7fb2c9ffb640 1 monclient: found mon.noname-c 2024-03-29T11:19:36.400 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.422+0000 7fb2c9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:36.400 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.422+0000 7fb2c9ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:19:36.400 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.422+0000 7fb2c9ffb640 1 -- 172.21.15.47:0/2922669192 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2cc14a950 con 0x7fb2cc0a9930 2024-03-29T11:19:36.400 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.422+0000 7fb2c9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:19:36.400 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.422+0000 7fb2c9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:19:36.401 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.422+0000 7fb2c9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:36.401 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.422+0000 7fb2c9ffb640 1 -- 172.21.15.47:0/2922669192 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fb2c4001690 con 0x7fb2cc0a9930 2024-03-29T11:19:36.401 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.422+0000 7fb2c9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:36.401 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.422+0000 7fb2c9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:36.401 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.422+0000 7fb2c9ffb640 10 monclient: dump: 2024-03-29T11:19:36.401 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:19:36.402 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:19:36.402 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:36.402 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:36.402 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:36.402 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:19:36.403 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:19:36.403 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:19:36.403 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:19:36.403 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:19:36.407 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.422+0000 7fb2c9ffb640 1 -- 172.21.15.47:0/2922669192 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2c40018b0 con 0x7fb2cc0a9930 2024-03-29T11:19:36.408 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.422+0000 7fb2c9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:36.408 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.422+0000 7fb2c9ffb640 1 -- 172.21.15.47:0/2922669192 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fb2c4001c40 con 0x7fb2cc0a9930 2024-03-29T11:19:36.408 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.422+0000 7fb2c9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:36.408 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.422+0000 7fb2c9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:36.408 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.422+0000 7fb2c9ffb640 10 monclient: dump: 2024-03-29T11:19:36.408 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:19:36.408 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:19:36.408 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:36.408 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:36.409 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:36.409 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:19:36.409 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:19:36.409 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:19:36.409 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:19:36.409 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:19:36.409 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.422+0000 7fb2d12d3640 5 monclient: authenticate success, global_id 4450 2024-03-29T11:19:36.409 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.422+0000 7fb2d12d3640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:36.409 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.422+0000 7fb2d12d3640 4 set_mon_vals no callback set 2024-03-29T11:19:36.409 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.423+0000 7fb2d12d3640 10 monclient: shutdown 2024-03-29T11:19:36.409 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.423+0000 7fb2d12d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:36.409 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.423+0000 7fb2d12d3640 1 -- 172.21.15.47:0/2922669192 >> v1:172.21.15.121:6789/0 conn(0x7fb2cc0a9930 legacy=0x7fb2cc143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:36.409 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.423+0000 7fb2d12d3640 1 -- 172.21.15.47:0/2922669192 shutdown_connections 2024-03-29T11:19:36.409 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.423+0000 7fb2d12d3640 1 -- 172.21.15.47:0/2922669192 >> 172.21.15.47:0/2922669192 conn(0x7fb2cc0b35e0 msgr2=0x7fb2cc0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:36.410 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.424+0000 7fb2d12d3640 1 -- 172.21.15.47:0/2922669192 shutdown_connections 2024-03-29T11:19:36.410 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.424+0000 7fb2d12d3640 1 -- 172.21.15.47:0/2922669192 wait complete. 2024-03-29T11:19:36.410 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.424+0000 7fb2d12d3640 10 monclient: build_initial_monmap 2024-03-29T11:19:36.410 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.424+0000 7fb2d12d3640 10 monclient: monmap: 2024-03-29T11:19:36.410 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:19:36.410 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:36.411 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:19:36.411 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:19:36.411 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:36.411 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:19:36.411 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:19:36.412 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:19:36.412 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:19:36.412 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:19:36.412 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.425+0000 7fb2d12d3640 1 librados: starting msgr at 2024-03-29T11:19:36.412 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.425+0000 7fb2d12d3640 1 librados: starting objecter 2024-03-29T11:19:36.412 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.425+0000 7fb2d12d3640 1 Processor -- start 2024-03-29T11:19:36.412 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.425+0000 7fb2d12d3640 1 -- start start 2024-03-29T11:19:36.413 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.425+0000 7fb2d12d3640 1 librados: setting wanted keys 2024-03-29T11:19:36.413 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.425+0000 7fb2d12d3640 1 librados: calling monclient init 2024-03-29T11:19:36.413 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.425+0000 7fb2d12d3640 10 monclient: init 2024-03-29T11:19:36.413 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.426+0000 7fb2d12d3640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:36.413 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.426+0000 7fb2d12d3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:36.413 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.426+0000 7fb2d12d3640 10 monclient(hunting): picked mon.noname-c con 0x7fb2cc0a9930 addr v1:172.21.15.47:6790/0 2024-03-29T11:19:36.413 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.426+0000 7fb2d12d3640 10 monclient(hunting): picked mon.noname-b con 0x7fb2cc149dc0 addr v1:172.21.15.121:6789/0 2024-03-29T11:19:36.413 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.426+0000 7fb2d12d3640 10 monclient(hunting): picked mon.noname-a con 0x7fb2cc14ff40 addr v1:172.21.15.47:6789/0 2024-03-29T11:19:36.413 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.426+0000 7fb2d12d3640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2cc1db880 con 0x7fb2cc14ff40 2024-03-29T11:19:36.413 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.426+0000 7fb2d12d3640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2cc147e90 con 0x7fb2cc149dc0 2024-03-29T11:19:36.413 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.426+0000 7fb2d12d3640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2cc148020 con 0x7fb2cc0a9930 2024-03-29T11:19:36.413 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.426+0000 7fb2d12d3640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:36.414 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.426+0000 7fb2d12d3640 10 monclient(hunting): authenticate will time out at 1219.614206s 2024-03-29T11:19:36.414 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.429+0000 7fb2ca7fc640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fb2cc149dc0 0x7fb2cc13f770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.47:51655/0 (socket says 172.21.15.47:51655) 2024-03-29T11:19:36.414 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.429+0000 7fb2ca7fc640 1 -- 172.21.15.47:0/3153013108 learned_addr learned my addr 172.21.15.47:0/3153013108 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:19:36.414 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.429+0000 7fb2b37fe640 10 client.?.objecter ms_handle_connect 0x7fb2cc149dc0 2024-03-29T11:19:36.414 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.429+0000 7fb2b37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:36.415 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.429+0000 7fb2b37fe640 1 -- 172.21.15.47:0/3153013108 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3763839664 0 0) 0x7fb2cc147e90 con 0x7fb2cc149dc0 2024-03-29T11:19:36.415 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.429+0000 7fb2b37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:36.415 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.429+0000 7fb2b37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:36.415 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.429+0000 7fb2b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:36.415 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.429+0000 7fb2b37fe640 10 monclient(hunting): my global_id is 4453 2024-03-29T11:19:36.415 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.430+0000 7fb2b37fe640 1 -- 172.21.15.47:0/3153013108 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2b8004eb0 con 0x7fb2cc149dc0 2024-03-29T11:19:36.415 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.430+0000 7fb2b37fe640 1 -- 172.21.15.47:0/3153013108 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2532096265 0 0) 0x7fb2b8004eb0 con 0x7fb2cc149dc0 2024-03-29T11:19:36.415 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.430+0000 7fb2b37fe640 1 -- 172.21.15.47:0/3153013108 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2cc147e90 con 0x7fb2cc149dc0 2024-03-29T11:19:36.415 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.430+0000 7fb2b37fe640 1 -- 172.21.15.47:0/3153013108 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fb2c00019d0 con 0x7fb2cc149dc0 2024-03-29T11:19:36.416 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.430+0000 7fb2b37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:36.416 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.430+0000 7fb2b37fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:36.416 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.430+0000 7fb2b37fe640 10 monclient(hunting): dump: 2024-03-29T11:19:36.416 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:19:36.416 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:19:36.416 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:36.416 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:36.416 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:36.416 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:19:36.416 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:19:36.416 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:19:36.416 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:19:36.417 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:19:36.417 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.430+0000 7fb2b37fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:36.417 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.430+0000 7fb2b37fe640 10 client.?.objecter ms_handle_connect 0x7fb2cc0a9930 2024-03-29T11:19:36.417 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.430+0000 7fb2b37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:36.417 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.430+0000 7fb2b37fe640 1 -- 172.21.15.47:0/3153013108 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1199197445 0 0) 0x7fb2cc147e90 con 0x7fb2cc149dc0 2024-03-29T11:19:36.417 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.430+0000 7fb2b37fe640 1 -- 172.21.15.47:0/3153013108 >> v1:172.21.15.47:6790/0 conn(0x7fb2cc0a9930 legacy=0x7fb2cc1dbd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:36.417 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.430+0000 7fb2b37fe640 1 -- 172.21.15.47:0/3153013108 >> v1:172.21.15.47:6789/0 conn(0x7fb2cc14ff40 legacy=0x7fb2cc1e6290 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:19:36.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.432+0000 7fb2b37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:19:36.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.432+0000 7fb2b37fe640 1 monclient: found mon.b 2024-03-29T11:19:36.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.432+0000 7fb2b37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:36.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.432+0000 7fb2b37fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:19:36.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.432+0000 7fb2b37fe640 1 -- 172.21.15.47:0/3153013108 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2cc1dc490 con 0x7fb2cc149dc0 2024-03-29T11:19:36.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.432+0000 7fb2b37fe640 10 monclient: _finish_auth 0 2024-03-29T11:19:36.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.432+0000 7fb2b37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:19:36.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.432+0000 7fb2b37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:36.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.434+0000 7fb2b37fe640 1 -- 172.21.15.47:0/3153013108 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2c0001cd0 con 0x7fb2cc149dc0 2024-03-29T11:19:36.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.434+0000 7fb2b37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:36.421 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.434+0000 7fb2c97fa640 4 set_mon_vals no callback set 2024-03-29T11:19:36.421 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.435+0000 7fb2b37fe640 1 -- 172.21.15.47:0/3153013108 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fb2c0003a60 con 0x7fb2cc149dc0 2024-03-29T11:19:36.421 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.435+0000 7fb2b37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:36.421 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.435+0000 7fb2b37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:36.421 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.435+0000 7fb2b37fe640 10 monclient: dump: 2024-03-29T11:19:36.421 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:19:36.421 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:19:36.421 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:36.421 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:36.421 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:36.421 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:19:36.421 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:19:36.421 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:19:36.421 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:19:36.422 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:19:36.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.435+0000 7fb2d12d3640 5 monclient: authenticate success, global_id 4453 2024-03-29T11:19:36.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.435+0000 7fb2d12d3640 10 monclient: _renew_subs 2024-03-29T11:19:36.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.435+0000 7fb2d12d3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:19:36.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.435+0000 7fb2d12d3640 1 -- 172.21.15.47:0/3153013108 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2cc1dc6a0 con 0x7fb2cc149dc0 2024-03-29T11:19:36.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.435+0000 7fb2d12d3640 10 client.4453.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:36.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.435+0000 7fb2d12d3640 10 monclient: _renew_subs 2024-03-29T11:19:36.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.435+0000 7fb2d12d3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:19:36.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.435+0000 7fb2d12d3640 1 -- 172.21.15.47:0/3153013108 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2cc140170 con 0x7fb2cc149dc0 2024-03-29T11:19:36.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.435+0000 7fb2d12d3640 1 librados: init done 2024-03-29T11:19:36.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.438+0000 7fb2b37fe640 1 -- 172.21.15.47:0/3153013108 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7fb2c0003bc0 con 0x7fb2cc149dc0 2024-03-29T11:19:36.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.438+0000 7fb2d12d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:36.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.438+0000 7fb2d12d3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:36.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.438+0000 7fb2d12d3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:19:36.423 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.438+0000 7fb2d12d3640 1 -- 172.21.15.47:0/3153013108 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2cc075f10 con 0x7fb2cc149dc0 2024-03-29T11:19:36.423 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.439+0000 7fb2b37fe640 1 -- 172.21.15.47:0/3153013108 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7fb2c0084ad0 con 0x7fb2cc149dc0 2024-03-29T11:19:36.423 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.439+0000 7fb2b37fe640 10 client.4453.objecter ms_dispatch 0x7fb2cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:36.423 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.439+0000 7fb2caffd640 10 monclient: get_auth_request con 0x7fb2b806db60 auth_method 0 2024-03-29T11:19:36.423 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.439+0000 7fb2b37fe640 3 client.4453.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:36.423 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.439+0000 7fb2b37fe640 3 client.4453.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:36.423 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.439+0000 7fb2b37fe640 20 client.4453.objecter dump_active .. 0 homeless 2024-03-29T11:19:36.423 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.439+0000 7fb2b37fe640 10 client.4453.objecter ms_handle_connect 0x7fb2b806db60 2024-03-29T11:19:36.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.449+0000 7fb2b37fe640 1 -- 172.21.15.47:0/3153013108 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb2c00503b0 con 0x7fb2cc149dc0 2024-03-29T11:19:36.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.449+0000 7fb2b37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:36.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.449+0000 7fb2b37fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:36.756 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.779+0000 7fb2d12d3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:36.757 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.779+0000 7fb2d12d3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:36.757 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.779+0000 7fb2d12d3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:19:36.757 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.779+0000 7fb2d12d3640 1 -- 172.21.15.47:0/3153013108 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb2cc06ab70 con 0x7fb2cc149dc0 2024-03-29T11:19:36.758 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.782+0000 7fb2b37fe640 1 -- 172.21.15.47:0/3153013108 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7fb2c00522e0 con 0x7fb2cc149dc0 2024-03-29T11:19:36.758 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.782+0000 7fb2b37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:36.758 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.782+0000 7fb2b37fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:36.759 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:19:36.759 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:36.762 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.786+0000 7fb2d12d3640 10 librados: watch_flush enter 2024-03-29T11:19:36.763 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.786+0000 7fb2d12d3640 10 librados: watch_flush exit 2024-03-29T11:19:36.763 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.786+0000 7fb2d12d3640 20 client.4453.objecter shutdown clearing up homeless session... 2024-03-29T11:19:36.763 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.786+0000 7fb2d12d3640 10 client.4453.objecter successfully canceled tick 2024-03-29T11:19:36.763 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.786+0000 7fb2d12d3640 1 -- 172.21.15.47:0/3153013108 >> v1:172.21.15.121:6816/1964117370 conn(0x7fb2b806db60 legacy=0x7fb2b8070020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:36.763 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.786+0000 7fb2d12d3640 10 monclient: shutdown 2024-03-29T11:19:36.763 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.786+0000 7fb2d12d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:36.763 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.786+0000 7fb2d12d3640 1 -- 172.21.15.47:0/3153013108 >> v1:172.21.15.121:6789/0 conn(0x7fb2cc149dc0 legacy=0x7fb2cc13f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:36.763 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.786+0000 7fb2d12d3640 1 -- 172.21.15.47:0/3153013108 shutdown_connections 2024-03-29T11:19:36.763 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.786+0000 7fb2d12d3640 1 -- 172.21.15.47:0/3153013108 >> 172.21.15.47:0/3153013108 conn(0x7fb2cc0b35e0 msgr2=0x7fb2cc0adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:36.764 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.786+0000 7fb2d12d3640 1 -- 172.21.15.47:0/3153013108 shutdown_connections 2024-03-29T11:19:36.764 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.786+0000 7fb2d12d3640 1 -- 172.21.15.47:0/3153013108 wait complete. 2024-03-29T11:19:36.764 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:36.786+0000 7fb2d12d3640 1 librados: shutdown 2024-03-29T11:19:36.781 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:43.588 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:43.738 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.761+0000 7f43865a3640 10 monclient: get_monmap_and_config 2024-03-29T11:19:43.739 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.761+0000 7f43865a3640 10 monclient: build_initial_monmap 2024-03-29T11:19:43.739 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.761+0000 7f43865a3640 10 monclient: monmap: 2024-03-29T11:19:43.739 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:19:43.739 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:19:43.739 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:19:43.762834+0000 2024-03-29T11:19:43.739 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:19:43.762834+0000 2024-03-29T11:19:43.739 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:43.739 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:19:43.739 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:19:43.739 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:19:43.739 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:19:43.739 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:19:43.740 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.763+0000 7f43865a3640 1 Processor -- start 2024-03-29T11:19:43.740 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.763+0000 7f43865a3640 1 -- start start 2024-03-29T11:19:43.740 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.763+0000 7f43865a3640 10 monclient: init 2024-03-29T11:19:43.741 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.764+0000 7f43865a3640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:43.741 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.764+0000 7f43865a3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:19:43.741 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.764+0000 7f43865a3640 10 monclient(hunting): picked mon.noname-b con 0x7f43801457f0 addr v1:172.21.15.47:6790/0 2024-03-29T11:19:43.741 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.764+0000 7f43865a3640 10 monclient(hunting): picked mon.noname-a con 0x7f438013f650 addr v1:172.21.15.47:6789/0 2024-03-29T11:19:43.742 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.764+0000 7f43865a3640 10 monclient(hunting): picked mon.noname-c con 0x7f43800a9d70 addr v1:172.21.15.121:6789/0 2024-03-29T11:19:43.742 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.764+0000 7f43865a3640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43800aaef0 con 0x7f438013f650 2024-03-29T11:19:43.742 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.764+0000 7f43865a3640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4380145bd0 con 0x7f43800a9d70 2024-03-29T11:19:43.742 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.764+0000 7f43865a3640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4380145eb0 con 0x7f43801457f0 2024-03-29T11:19:43.742 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.764+0000 7f43865a3640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:43.742 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.764+0000 7f43865a3640 10 monclient(hunting): authenticate will time out at 1226.952361s 2024-03-29T11:19:43.745 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.766+0000 7f437ffff640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f43801457f0 0x7f438013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:52246/0 (socket says 172.21.15.47:52246) 2024-03-29T11:19:43.745 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.766+0000 7f437ffff640 1 -- 172.21.15.47:0/640736976 learned_addr learned my addr 172.21.15.47:0/640736976 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:19:43.745 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.767+0000 7f437effd640 1 -- 172.21.15.47:0/640736976 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 436533585 0 0) 0x7f43800aaef0 con 0x7f438013f650 2024-03-29T11:19:43.745 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.767+0000 7f437effd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:43.745 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.767+0000 7f437effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:43.745 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.767+0000 7f437effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:43.745 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.767+0000 7f437effd640 10 monclient(hunting): my global_id is 4449 2024-03-29T11:19:43.745 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.767+0000 7f437effd640 1 -- 172.21.15.47:0/640736976 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4368003520 con 0x7f438013f650 2024-03-29T11:19:43.745 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.768+0000 7f437effd640 1 -- 172.21.15.47:0/640736976 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 605677108 0 0) 0x7f4380145eb0 con 0x7f43801457f0 2024-03-29T11:19:43.745 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.768+0000 7f437effd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:43.745 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.768+0000 7f437effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:43.746 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.768+0000 7f437effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:43.746 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.768+0000 7f437effd640 10 monclient(hunting): my global_id is 4502 2024-03-29T11:19:43.746 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.768+0000 7f437effd640 1 -- 172.21.15.47:0/640736976 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43800aaef0 con 0x7f43801457f0 2024-03-29T11:19:43.746 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.769+0000 7f437effd640 1 -- 172.21.15.47:0/640736976 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1696556768 0 0) 0x7f4368003520 con 0x7f438013f650 2024-03-29T11:19:43.746 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.769+0000 7f437effd640 1 -- 172.21.15.47:0/640736976 >> v1:172.21.15.47:6790/0 conn(0x7f43801457f0 legacy=0x7f438013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:43.746 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.769+0000 7f437effd640 1 -- 172.21.15.47:0/640736976 >> v1:172.21.15.121:6789/0 conn(0x7f43800a9d70 legacy=0x7f438014d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:19:43.750 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.770+0000 7f437effd640 10 monclient: _finish_hunting 0 2024-03-29T11:19:43.750 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.770+0000 7f437effd640 1 monclient: found mon.noname-a 2024-03-29T11:19:43.750 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.770+0000 7f437effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:43.750 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.770+0000 7f437effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-29T11:19:43.750 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.770+0000 7f437effd640 1 -- 172.21.15.47:0/640736976 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4380146340 con 0x7f438013f650 2024-03-29T11:19:43.750 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.770+0000 7f437effd640 10 monclient: _finish_auth 0 2024-03-29T11:19:43.750 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.770+0000 7f437effd640 10 monclient: _check_auth_tickets 2024-03-29T11:19:43.750 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.770+0000 7f437effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:43.750 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.770+0000 7f43865a3640 5 monclient: authenticate success, global_id 4449 2024-03-29T11:19:43.750 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.770+0000 7f43865a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:43.750 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.770+0000 7f437effd640 1 -- 172.21.15.47:0/640736976 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f4374001690 con 0x7f438013f650 2024-03-29T11:19:43.750 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.770+0000 7f437effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:43.750 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.770+0000 7f437effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:43.751 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.770+0000 7f437effd640 10 monclient: dump: 2024-03-29T11:19:43.751 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:19:43.751 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:19:43.751 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:43.751 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:43.751 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:43.751 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:19:43.751 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:19:43.751 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:19:43.751 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:19:43.751 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:19:43.751 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.770+0000 7f43865a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:43.751 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.770+0000 7f437effd640 1 -- 172.21.15.47:0/640736976 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f43740018b0 con 0x7f438013f650 2024-03-29T11:19:43.751 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.770+0000 7f437effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:43.751 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.770+0000 7f43865a3640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:43.752 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.770+0000 7f43865a3640 4 set_mon_vals no callback set 2024-03-29T11:19:43.752 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.771+0000 7f437effd640 1 -- 172.21.15.47:0/640736976 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f4374001c40 con 0x7f438013f650 2024-03-29T11:19:43.752 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.771+0000 7f437effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:43.752 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.771+0000 7f437effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:43.752 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.771+0000 7f437effd640 10 monclient: dump: 2024-03-29T11:19:43.752 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:19:43.752 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:19:43.752 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:43.752 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:43.752 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:43.752 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:19:43.752 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:19:43.752 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:19:43.752 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:19:43.752 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:19:43.752 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.771+0000 7f43865a3640 10 monclient: shutdown 2024-03-29T11:19:43.753 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.771+0000 7f43865a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:43.753 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.771+0000 7f43865a3640 1 -- 172.21.15.47:0/640736976 >> v1:172.21.15.47:6789/0 conn(0x7f438013f650 legacy=0x7f43800b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:43.753 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.772+0000 7f43865a3640 1 -- 172.21.15.47:0/640736976 shutdown_connections 2024-03-29T11:19:43.753 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.772+0000 7f43865a3640 1 -- 172.21.15.47:0/640736976 >> 172.21.15.47:0/640736976 conn(0x7f4380066a00 msgr2=0x7f4380066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:43.753 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.772+0000 7f43865a3640 1 -- 172.21.15.47:0/640736976 shutdown_connections 2024-03-29T11:19:43.753 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.772+0000 7f43865a3640 1 -- 172.21.15.47:0/640736976 wait complete. 2024-03-29T11:19:43.756 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.773+0000 7f43865a3640 10 monclient: build_initial_monmap 2024-03-29T11:19:43.756 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.773+0000 7f43865a3640 10 monclient: monmap: 2024-03-29T11:19:43.756 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:19:43.756 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:43.756 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:19:43.756 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:19:43.756 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:43.756 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:19:43.757 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:19:43.757 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:19:43.757 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:19:43.757 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:19:43.757 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.774+0000 7f43865a3640 1 librados: starting msgr at 2024-03-29T11:19:43.757 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.774+0000 7f43865a3640 1 librados: starting objecter 2024-03-29T11:19:43.757 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.774+0000 7f43865a3640 1 Processor -- start 2024-03-29T11:19:43.757 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.774+0000 7f43865a3640 1 -- start start 2024-03-29T11:19:43.757 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.774+0000 7f43865a3640 1 librados: setting wanted keys 2024-03-29T11:19:43.757 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.774+0000 7f43865a3640 1 librados: calling monclient init 2024-03-29T11:19:43.757 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.774+0000 7f43865a3640 10 monclient: init 2024-03-29T11:19:43.757 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.775+0000 7f43865a3640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:43.758 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.775+0000 7f43865a3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:43.758 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.775+0000 7f43865a3640 10 monclient(hunting): picked mon.noname-b con 0x7f43800a9d70 addr v1:172.21.15.121:6789/0 2024-03-29T11:19:43.758 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.775+0000 7f43865a3640 10 monclient(hunting): picked mon.noname-a con 0x7f438013f650 addr v1:172.21.15.47:6789/0 2024-03-29T11:19:43.758 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.775+0000 7f43865a3640 10 monclient(hunting): picked mon.noname-c con 0x7f43801457f0 addr v1:172.21.15.47:6790/0 2024-03-29T11:19:43.758 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.775+0000 7f4384b40640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f43801457f0 0x7f4380137810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:53248/0 (socket says 172.21.15.47:53248) 2024-03-29T11:19:43.758 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.775+0000 7f4384b40640 1 -- 172.21.15.47:0/1319149314 learned_addr learned my addr 172.21.15.47:0/1319149314 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:19:43.758 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.775+0000 7f43865a3640 1 -- 172.21.15.47:0/1319149314 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f438013af00 con 0x7f438013f650 2024-03-29T11:19:43.758 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.775+0000 7f43865a3640 1 -- 172.21.15.47:0/1319149314 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4380137f20 con 0x7f43800a9d70 2024-03-29T11:19:43.758 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.775+0000 7f43865a3640 1 -- 172.21.15.47:0/1319149314 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f438013b150 con 0x7f43801457f0 2024-03-29T11:19:43.758 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.775+0000 7f43865a3640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:43.758 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.775+0000 7f43865a3640 10 monclient(hunting): authenticate will time out at 1226.963766s 2024-03-29T11:19:43.758 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.775+0000 7f437cff9640 10 client.?.objecter ms_handle_connect 0x7f43801457f0 2024-03-29T11:19:43.758 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.775+0000 7f437cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:43.759 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.776+0000 7f437cff9640 1 -- 172.21.15.47:0/1319149314 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3452387057 0 0) 0x7f438013b150 con 0x7f43801457f0 2024-03-29T11:19:43.759 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.776+0000 7f437cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:19:43.759 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.776+0000 7f437cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:43.759 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.776+0000 7f437cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:43.759 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.776+0000 7f437cff9640 10 monclient(hunting): my global_id is 4505 2024-03-29T11:19:43.759 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.776+0000 7f437cff9640 1 -- 172.21.15.47:0/1319149314 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f436c004eb0 con 0x7f43801457f0 2024-03-29T11:19:43.759 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.777+0000 7f437cff9640 1 -- 172.21.15.47:0/1319149314 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3909233206 0 0) 0x7f436c004eb0 con 0x7f43801457f0 2024-03-29T11:19:43.759 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.777+0000 7f437cff9640 1 -- 172.21.15.47:0/1319149314 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f438013b150 con 0x7f43801457f0 2024-03-29T11:19:43.759 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.777+0000 7f437cff9640 1 -- 172.21.15.47:0/1319149314 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f43700017f0 con 0x7f43801457f0 2024-03-29T11:19:43.759 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.777+0000 7f437cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:43.759 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.777+0000 7f437cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:43.759 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.777+0000 7f437cff9640 10 monclient(hunting): dump: 2024-03-29T11:19:43.760 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:19:43.760 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:19:43.760 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:43.760 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:43.760 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:43.760 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:19:43.760 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:19:43.760 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:19:43.760 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:19:43.760 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:19:43.760 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.777+0000 7f437cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:43.760 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.779+0000 7f437cff9640 10 client.?.objecter ms_handle_connect 0x7f438013f650 2024-03-29T11:19:43.760 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.779+0000 7f437cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:43.760 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.779+0000 7f437cff9640 1 -- 172.21.15.47:0/1319149314 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2544623121 0 0) 0x7f438013af00 con 0x7f438013f650 2024-03-29T11:19:43.760 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.779+0000 7f437cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:19:43.761 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.779+0000 7f437cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:43.761 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.779+0000 7f437cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:43.761 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.779+0000 7f437cff9640 10 monclient(hunting): my global_id is 4452 2024-03-29T11:19:43.761 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.779+0000 7f437cff9640 1 -- 172.21.15.47:0/1319149314 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f436c004eb0 con 0x7f438013f650 2024-03-29T11:19:43.762 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.780+0000 7f437cff9640 1 -- 172.21.15.47:0/1319149314 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2444075794 0 0) 0x7f438013b150 con 0x7f43801457f0 2024-03-29T11:19:43.762 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.780+0000 7f437cff9640 1 -- 172.21.15.47:0/1319149314 >> v1:172.21.15.121:6789/0 conn(0x7f43800a9d70 legacy=0x7f43801db830 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:19:43.762 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.780+0000 7f437cff9640 1 -- 172.21.15.47:0/1319149314 >> v1:172.21.15.47:6789/0 conn(0x7f438013f650 legacy=0x7f4380137100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:43.762 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.780+0000 7f437cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:19:43.762 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.780+0000 7f437cff9640 1 monclient: found mon.c 2024-03-29T11:19:43.762 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.780+0000 7f437cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:43.763 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.780+0000 7f437cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:19:43.763 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.780+0000 7f437cff9640 1 -- 172.21.15.47:0/1319149314 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4380141f00 con 0x7f43801457f0 2024-03-29T11:19:43.763 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.780+0000 7f437cff9640 10 monclient: _finish_auth 0 2024-03-29T11:19:43.763 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.780+0000 7f437cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:19:43.763 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.780+0000 7f437cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:43.763 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.780+0000 7f43865a3640 5 monclient: authenticate success, global_id 4505 2024-03-29T11:19:43.763 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.780+0000 7f43865a3640 10 monclient: _renew_subs 2024-03-29T11:19:43.763 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.780+0000 7f43865a3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:19:43.763 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.780+0000 7f43865a3640 1 -- 172.21.15.47:0/1319149314 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4380142090 con 0x7f43801457f0 2024-03-29T11:19:43.763 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.780+0000 7f43865a3640 10 client.4505.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:43.763 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.780+0000 7f43865a3640 10 monclient: _renew_subs 2024-03-29T11:19:43.763 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.780+0000 7f43865a3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:19:43.763 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.780+0000 7f43865a3640 1 -- 172.21.15.47:0/1319149314 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4380142cb0 con 0x7f43801457f0 2024-03-29T11:19:43.763 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.780+0000 7f43865a3640 1 librados: init done 2024-03-29T11:19:43.764 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.783+0000 7f43865a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:43.764 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.783+0000 7f43865a3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:43.764 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.783+0000 7f43865a3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:19:43.764 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.783+0000 7f43865a3640 1 -- 172.21.15.47:0/1319149314 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4380075f10 con 0x7f43801457f0 2024-03-29T11:19:43.764 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.784+0000 7f437cff9640 1 -- 172.21.15.47:0/1319149314 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4370001dc0 con 0x7f43801457f0 2024-03-29T11:19:43.764 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.784+0000 7f437cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:43.764 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.784+0000 7f437cff9640 1 -- 172.21.15.47:0/1319149314 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f4370004aa0 con 0x7f43801457f0 2024-03-29T11:19:43.764 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.784+0000 7f437cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:43.764 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.784+0000 7f437cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:43.764 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.784+0000 7f437cff9640 10 monclient: dump: 2024-03-29T11:19:43.764 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:19:43.765 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:19:43.765 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:43.765 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:43.765 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:43.765 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:19:43.765 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:19:43.765 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:19:43.765 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:19:43.765 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:19:43.765 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.784+0000 7f437cff9640 1 -- 172.21.15.47:0/1319149314 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7f437001aaf0 con 0x7f43801457f0 2024-03-29T11:19:43.765 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.784+0000 7f437e7fc640 4 set_mon_vals no callback set 2024-03-29T11:19:43.766 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.787+0000 7f437cff9640 1 -- 172.21.15.47:0/1319149314 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7f4370084b20 con 0x7f43801457f0 2024-03-29T11:19:43.766 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.787+0000 7f437cff9640 10 client.4505.objecter ms_dispatch 0x7f4380000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:43.766 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.787+0000 7f437cff9640 3 client.4505.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:43.766 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.787+0000 7f437cff9640 3 client.4505.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:43.766 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.787+0000 7f437ffff640 10 monclient: get_auth_request con 0x7f436c06de00 auth_method 0 2024-03-29T11:19:43.766 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.787+0000 7f437cff9640 20 client.4505.objecter dump_active .. 0 homeless 2024-03-29T11:19:43.768 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.791+0000 7f437cff9640 10 client.4505.objecter ms_handle_connect 0x7f436c06de00 2024-03-29T11:19:43.772 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.792+0000 7f437cff9640 1 -- 172.21.15.47:0/1319149314 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4370050480 con 0x7f43801457f0 2024-03-29T11:19:43.772 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.792+0000 7f437cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:43.772 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:43.792+0000 7f437cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:44.102 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:44.124+0000 7f43865a3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:44.102 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:44.124+0000 7f43865a3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:44.102 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:44.124+0000 7f43865a3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:19:44.102 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:44.124+0000 7f43865a3640 1 -- 172.21.15.47:0/1319149314 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4380144770 con 0x7f43801457f0 2024-03-29T11:19:44.104 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:44.127+0000 7f437cff9640 1 -- 172.21.15.47:0/1319149314 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7f43700523b0 con 0x7f43801457f0 2024-03-29T11:19:44.104 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:44.127+0000 7f437cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:44.104 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:44.127+0000 7f437cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:44.105 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:19:44.105 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:44.108 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:44.131+0000 7f43865a3640 10 librados: watch_flush enter 2024-03-29T11:19:44.108 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:44.131+0000 7f43865a3640 10 librados: watch_flush exit 2024-03-29T11:19:44.108 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:44.131+0000 7f43865a3640 20 client.4505.objecter shutdown clearing up homeless session... 2024-03-29T11:19:44.108 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:44.131+0000 7f43865a3640 10 client.4505.objecter successfully canceled tick 2024-03-29T11:19:44.110 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:44.133+0000 7f43865a3640 1 -- 172.21.15.47:0/1319149314 >> v1:172.21.15.121:6816/1964117370 conn(0x7f436c06de00 legacy=0x7f436c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:44.110 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:44.133+0000 7f43865a3640 10 monclient: shutdown 2024-03-29T11:19:44.110 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:44.133+0000 7f43865a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:44.110 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:44.133+0000 7f43865a3640 1 -- 172.21.15.47:0/1319149314 >> v1:172.21.15.47:6790/0 conn(0x7f43801457f0 legacy=0x7f4380137810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:44.110 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:44.133+0000 7f43865a3640 1 -- 172.21.15.47:0/1319149314 shutdown_connections 2024-03-29T11:19:44.110 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:44.133+0000 7f43865a3640 1 -- 172.21.15.47:0/1319149314 >> 172.21.15.47:0/1319149314 conn(0x7f4380066a00 msgr2=0x7f43800b4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:44.110 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:44.133+0000 7f43865a3640 1 -- 172.21.15.47:0/1319149314 shutdown_connections 2024-03-29T11:19:44.111 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:44.133+0000 7f43865a3640 1 -- 172.21.15.47:0/1319149314 wait complete. 2024-03-29T11:19:44.111 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:44.134+0000 7f43865a3640 1 librados: shutdown 2024-03-29T11:19:44.128 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:50.936 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:51.090 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.113+0000 7f241715d640 10 monclient: get_monmap_and_config 2024-03-29T11:19:51.091 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.113+0000 7f241715d640 10 monclient: build_initial_monmap 2024-03-29T11:19:51.091 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.113+0000 7f241715d640 10 monclient: monmap: 2024-03-29T11:19:51.091 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:19:51.091 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:19:51.091 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:19:51.114122+0000 2024-03-29T11:19:51.091 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:19:51.114122+0000 2024-03-29T11:19:51.091 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:51.091 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:19:51.091 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:19:51.091 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:19:51.091 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:19:51.091 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:19:51.091 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.114+0000 7f241715d640 1 Processor -- start 2024-03-29T11:19:51.092 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.114+0000 7f241715d640 1 -- start start 2024-03-29T11:19:51.092 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.114+0000 7f241715d640 10 monclient: init 2024-03-29T11:19:51.092 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.115+0000 7f241715d640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:51.092 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.115+0000 7f241715d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:19:51.092 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.115+0000 7f241715d640 10 monclient(hunting): picked mon.noname-b con 0x7f24100a9ac0 addr v1:172.21.15.47:6790/0 2024-03-29T11:19:51.092 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.115+0000 7f241715d640 10 monclient(hunting): picked mon.noname-a con 0x7f241013ee90 addr v1:172.21.15.47:6789/0 2024-03-29T11:19:51.094 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.117+0000 7f241715d640 10 monclient(hunting): picked mon.noname-c con 0x7f24100af900 addr v1:172.21.15.121:6789/0 2024-03-29T11:19:51.094 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.117+0000 7f241715d640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24100afd10 con 0x7f241013ee90 2024-03-29T11:19:51.094 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.117+0000 7f241715d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24100af6f0 con 0x7f24100af900 2024-03-29T11:19:51.094 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.117+0000 7f241715d640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24100aac80 con 0x7f24100a9ac0 2024-03-29T11:19:51.094 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.117+0000 7f241715d640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:51.094 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.117+0000 7f241715d640 10 monclient(hunting): authenticate will time out at 1234.305197s 2024-03-29T11:19:51.094 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.117+0000 7f2414ef9640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f24100a9ac0 0x7f24100adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:36182/0 (socket says 172.21.15.47:36182) 2024-03-29T11:19:51.095 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.117+0000 7f2414ef9640 1 -- 172.21.15.47:0/268670576 learned_addr learned my addr 172.21.15.47:0/268670576 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:19:51.096 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.118+0000 7f240f7fe640 1 -- 172.21.15.47:0/268670576 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2499413171 0 0) 0x7f24100aac80 con 0x7f24100a9ac0 2024-03-29T11:19:51.096 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.118+0000 7f240f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:51.096 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.118+0000 7f240f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:51.096 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.118+0000 7f240f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:51.096 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.118+0000 7f240f7fe640 10 monclient(hunting): my global_id is 4508 2024-03-29T11:19:51.096 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.118+0000 7f240f7fe640 1 -- 172.21.15.47:0/268670576 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23f8003520 con 0x7f24100a9ac0 2024-03-29T11:19:51.096 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.118+0000 7f240f7fe640 1 -- 172.21.15.47:0/268670576 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3446407283 0 0) 0x7f23f8003520 con 0x7f24100a9ac0 2024-03-29T11:19:51.096 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.118+0000 7f240f7fe640 1 -- 172.21.15.47:0/268670576 >> v1:172.21.15.121:6789/0 conn(0x7f24100af900 legacy=0x7f2410149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:19:51.096 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.118+0000 7f240f7fe640 1 -- 172.21.15.47:0/268670576 >> v1:172.21.15.47:6789/0 conn(0x7f241013ee90 legacy=0x7f241013b5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:19:51.098 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.121+0000 7f240f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:19:51.098 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.121+0000 7f240f7fe640 1 monclient: found mon.noname-b 2024-03-29T11:19:51.098 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.121+0000 7f240f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:51.098 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.121+0000 7f240f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-29T11:19:51.099 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.121+0000 7f240f7fe640 1 -- 172.21.15.47:0/268670576 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24100aaf60 con 0x7f24100a9ac0 2024-03-29T11:19:51.099 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.121+0000 7f240f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:19:51.099 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.121+0000 7f240f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:19:51.099 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.121+0000 7f240f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:51.099 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.121+0000 7f240f7fe640 1 -- 172.21.15.47:0/268670576 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f2400001690 con 0x7f24100a9ac0 2024-03-29T11:19:51.099 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.121+0000 7f241715d640 5 monclient: authenticate success, global_id 4508 2024-03-29T11:19:51.099 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.122+0000 7f241715d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:51.099 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.122+0000 7f240f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:51.099 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.122+0000 7f240f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:51.099 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.122+0000 7f240f7fe640 10 monclient: dump: 2024-03-29T11:19:51.099 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:19:51.099 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:19:51.099 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:51.099 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:51.100 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:51.100 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:19:51.100 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:19:51.100 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:19:51.100 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:19:51.100 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:19:51.100 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.122+0000 7f241715d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:51.101 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.122+0000 7f240f7fe640 1 -- 172.21.15.47:0/268670576 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2400001920 con 0x7f24100a9ac0 2024-03-29T11:19:51.101 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.122+0000 7f240f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:51.102 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.122+0000 7f240f7fe640 1 -- 172.21.15.47:0/268670576 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f2400001cb0 con 0x7f24100a9ac0 2024-03-29T11:19:51.102 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.122+0000 7f241715d640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:51.102 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.122+0000 7f241715d640 4 set_mon_vals no callback set 2024-03-29T11:19:51.102 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.123+0000 7f241715d640 10 monclient: shutdown 2024-03-29T11:19:51.102 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.123+0000 7f240f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:51.102 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.123+0000 7f240f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:51.102 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.123+0000 7f240f7fe640 10 monclient: dump: 2024-03-29T11:19:51.102 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:19:51.102 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:19:51.102 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:51.102 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:51.102 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:51.102 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:19:51.102 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:19:51.103 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:19:51.103 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:19:51.103 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:19:51.103 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.123+0000 7f241715d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:51.103 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.123+0000 7f241715d640 1 -- 172.21.15.47:0/268670576 >> v1:172.21.15.47:6790/0 conn(0x7f24100a9ac0 legacy=0x7f24100adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:51.103 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.123+0000 7f241715d640 1 -- 172.21.15.47:0/268670576 shutdown_connections 2024-03-29T11:19:51.103 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.123+0000 7f241715d640 1 -- 172.21.15.47:0/268670576 >> 172.21.15.47:0/268670576 conn(0x7f24100b35e0 msgr2=0x7f24100b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:51.103 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.123+0000 7f241715d640 1 -- 172.21.15.47:0/268670576 shutdown_connections 2024-03-29T11:19:51.103 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.123+0000 7f241715d640 1 -- 172.21.15.47:0/268670576 wait complete. 2024-03-29T11:19:51.103 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.125+0000 7f241715d640 10 monclient: build_initial_monmap 2024-03-29T11:19:51.103 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.125+0000 7f241715d640 10 monclient: monmap: 2024-03-29T11:19:51.103 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:19:51.103 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:51.103 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:19:51.103 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:19:51.103 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:51.104 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:19:51.104 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:19:51.104 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:19:51.104 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:19:51.104 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:19:51.104 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.125+0000 7f241715d640 1 librados: starting msgr at 2024-03-29T11:19:51.105 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.125+0000 7f241715d640 1 librados: starting objecter 2024-03-29T11:19:51.105 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.126+0000 7f241715d640 1 Processor -- start 2024-03-29T11:19:51.105 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.127+0000 7f241715d640 1 -- start start 2024-03-29T11:19:51.105 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.127+0000 7f241715d640 1 librados: setting wanted keys 2024-03-29T11:19:51.105 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.127+0000 7f241715d640 1 librados: calling monclient init 2024-03-29T11:19:51.105 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.127+0000 7f241715d640 10 monclient: init 2024-03-29T11:19:51.105 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.128+0000 7f241715d640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:51.108 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.132+0000 7f241715d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:19:51.110 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.132+0000 7f241715d640 10 monclient(hunting): picked mon.noname-a con 0x7f24100a9ac0 addr v1:172.21.15.47:6789/0 2024-03-29T11:19:51.110 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.133+0000 7f241715d640 10 monclient(hunting): picked mon.noname-b con 0x7f24100af900 addr v1:172.21.15.121:6789/0 2024-03-29T11:19:51.110 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.133+0000 7f241715d640 10 monclient(hunting): picked mon.noname-c con 0x7f241013ee90 addr v1:172.21.15.47:6790/0 2024-03-29T11:19:51.110 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.133+0000 7f241715d640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24101425d0 con 0x7f24100a9ac0 2024-03-29T11:19:51.110 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.133+0000 7f241715d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2410070990 con 0x7f24100af900 2024-03-29T11:19:51.110 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.133+0000 7f241715d640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f241006f7a0 con 0x7f241013ee90 2024-03-29T11:19:51.110 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.133+0000 7f241715d640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:51.110 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.133+0000 7f241715d640 10 monclient(hunting): authenticate will time out at 1234.321105s 2024-03-29T11:19:51.113 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.136+0000 7f24156fa640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f241013ee90 0x7f24101d3990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:47916/0 (socket says 172.21.15.47:47916) 2024-03-29T11:19:51.113 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.136+0000 7f24156fa640 1 -- 172.21.15.47:0/2834540928 learned_addr learned my addr 172.21.15.47:0/2834540928 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:19:51.113 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.136+0000 7f240d7fa640 10 client.?.objecter ms_handle_connect 0x7f241013ee90 2024-03-29T11:19:51.113 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.136+0000 7f240d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:51.114 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.137+0000 7f240d7fa640 10 client.?.objecter ms_handle_connect 0x7f24100a9ac0 2024-03-29T11:19:51.115 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.137+0000 7f240d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:51.115 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.137+0000 7f240d7fa640 1 -- 172.21.15.47:0/2834540928 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 75971392 0 0) 0x7f24101425d0 con 0x7f24100a9ac0 2024-03-29T11:19:51.115 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.137+0000 7f240d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:19:51.115 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.137+0000 7f240d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:51.115 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.137+0000 7f240d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:51.115 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.137+0000 7f240d7fa640 10 monclient(hunting): my global_id is 4458 2024-03-29T11:19:51.115 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.137+0000 7f240d7fa640 1 -- 172.21.15.47:0/2834540928 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23fc004eb0 con 0x7f24100a9ac0 2024-03-29T11:19:51.115 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.137+0000 7f240d7fa640 10 client.?.objecter ms_handle_connect 0x7f24100af900 2024-03-29T11:19:51.115 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.138+0000 7f240d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:51.115 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.138+0000 7f240d7fa640 1 -- 172.21.15.47:0/2834540928 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 77858175 0 0) 0x7f23fc004eb0 con 0x7f24100a9ac0 2024-03-29T11:19:51.118 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.138+0000 7f240d7fa640 1 -- 172.21.15.47:0/2834540928 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24101425d0 con 0x7f24100a9ac0 2024-03-29T11:19:51.118 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.138+0000 7f240d7fa640 1 -- 172.21.15.47:0/2834540928 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f2400001550 con 0x7f24100a9ac0 2024-03-29T11:19:51.118 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.138+0000 7f240d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:51.118 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.138+0000 7f240d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:51.119 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.138+0000 7f240d7fa640 10 monclient(hunting): dump: 2024-03-29T11:19:51.119 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:19:51.119 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:19:51.119 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:51.119 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:51.119 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:51.119 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:19:51.119 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:19:51.119 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:19:51.119 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:19:51.119 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:19:51.119 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.138+0000 7f240d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:51.119 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.138+0000 7f240d7fa640 1 -- 172.21.15.47:0/2834540928 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1086714256 0 0) 0x7f24101425d0 con 0x7f24100a9ac0 2024-03-29T11:19:51.119 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.138+0000 7f240d7fa640 1 -- 172.21.15.47:0/2834540928 >> v1:172.21.15.47:6790/0 conn(0x7f241013ee90 legacy=0x7f24101d3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:51.119 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.138+0000 7f240d7fa640 1 -- 172.21.15.47:0/2834540928 >> v1:172.21.15.121:6789/0 conn(0x7f24100af900 legacy=0x7f24101d3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:51.119 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.138+0000 7f240d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:19:51.120 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.138+0000 7f240d7fa640 1 monclient: found mon.a 2024-03-29T11:19:51.120 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.138+0000 7f240d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:51.120 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.138+0000 7f240d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:19:51.120 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.138+0000 7f240d7fa640 1 -- 172.21.15.47:0/2834540928 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f241006f930 con 0x7f24100a9ac0 2024-03-29T11:19:51.120 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.138+0000 7f240d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:19:51.120 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.138+0000 7f240d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:19:51.120 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.138+0000 7f240d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:51.120 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.138+0000 7f241715d640 5 monclient: authenticate success, global_id 4458 2024-03-29T11:19:51.120 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.138+0000 7f241715d640 10 monclient: _renew_subs 2024-03-29T11:19:51.120 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.138+0000 7f241715d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:19:51.120 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.138+0000 7f241715d640 1 -- 172.21.15.47:0/2834540928 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2410072400 con 0x7f24100a9ac0 2024-03-29T11:19:51.120 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.140+0000 7f240d7fa640 1 -- 172.21.15.47:0/2834540928 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24000017b0 con 0x7f24100a9ac0 2024-03-29T11:19:51.120 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.140+0000 7f240d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:51.120 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.140+0000 7f240d7fa640 1 -- 172.21.15.47:0/2834540928 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f2400004d90 con 0x7f24100a9ac0 2024-03-29T11:19:51.120 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.140+0000 7f240d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:51.120 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.140+0000 7f240d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:51.121 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.140+0000 7f240d7fa640 10 monclient: dump: 2024-03-29T11:19:51.121 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:19:51.121 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:19:51.121 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:51.121 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:51.121 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:51.121 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:19:51.121 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:19:51.121 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:19:51.121 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:19:51.121 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:19:51.121 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.141+0000 7f241715d640 10 client.4458.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:51.121 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.141+0000 7f241715d640 10 monclient: _renew_subs 2024-03-29T11:19:51.121 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.141+0000 7f241715d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:19:51.121 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.141+0000 7f241715d640 1 -- 172.21.15.47:0/2834540928 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f24101decb0 con 0x7f24100a9ac0 2024-03-29T11:19:51.121 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.141+0000 7f241715d640 1 librados: init done 2024-03-29T11:19:51.121 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.140+0000 7f240effd640 4 set_mon_vals no callback set 2024-03-29T11:19:51.122 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.143+0000 7f23f6ffd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:51.122 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.143+0000 7f23f6ffd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:51.123 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.143+0000 7f23f6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:19:51.123 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.143+0000 7f23f6ffd640 1 -- 172.21.15.47:0/2834540928 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2410075f10 con 0x7f24100a9ac0 2024-03-29T11:19:51.123 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.144+0000 7f240d7fa640 1 -- 172.21.15.47:0/2834540928 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7f240001ada0 con 0x7f24100a9ac0 2024-03-29T11:19:51.123 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.145+0000 7f240d7fa640 1 -- 172.21.15.47:0/2834540928 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7f2400054980 con 0x7f24100a9ac0 2024-03-29T11:19:51.123 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.145+0000 7f240d7fa640 10 client.4458.objecter ms_dispatch 0x7f2410000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:51.123 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.145+0000 7f240d7fa640 3 client.4458.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:51.123 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.145+0000 7f240d7fa640 3 client.4458.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:51.123 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.145+0000 7f240d7fa640 20 client.4458.objecter dump_active .. 0 homeless 2024-03-29T11:19:51.123 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.146+0000 7f240ffff640 10 monclient: get_auth_request con 0x7f23fc0720f0 auth_method 0 2024-03-29T11:19:51.124 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.147+0000 7f240d7fa640 10 client.4458.objecter ms_handle_connect 0x7f23fc0720f0 2024-03-29T11:19:51.129 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.152+0000 7f240d7fa640 1 -- 172.21.15.47:0/2834540928 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2400050490 con 0x7f24100a9ac0 2024-03-29T11:19:51.130 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.152+0000 7f240d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:51.130 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.152+0000 7f240d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:51.463 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.486+0000 7f23f6ffd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:51.463 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.486+0000 7f23f6ffd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:51.463 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.486+0000 7f23f6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:19:51.463 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.486+0000 7f23f6ffd640 1 -- 172.21.15.47:0/2834540928 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f24100ac240 con 0x7f24100a9ac0 2024-03-29T11:19:51.464 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.488+0000 7f240d7fa640 1 -- 172.21.15.47:0/2834540928 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7f2400050490 con 0x7f24100a9ac0 2024-03-29T11:19:51.464 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.488+0000 7f240d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:51.465 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.488+0000 7f240d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:51.470 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:19:51.470 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:51.471 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.491+0000 7f23f6ffd640 10 librados: watch_flush enter 2024-03-29T11:19:51.471 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.492+0000 7f23f6ffd640 10 librados: watch_flush exit 2024-03-29T11:19:51.471 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.492+0000 7f23f6ffd640 20 client.4458.objecter shutdown clearing up homeless session... 2024-03-29T11:19:51.471 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.492+0000 7f23f6ffd640 10 client.4458.objecter successfully canceled tick 2024-03-29T11:19:51.471 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.492+0000 7f23f6ffd640 1 -- 172.21.15.47:0/2834540928 >> v1:172.21.15.121:6816/1964117370 conn(0x7f23fc0720f0 legacy=0x7f23fc0745b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:51.471 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.492+0000 7f23f6ffd640 10 monclient: shutdown 2024-03-29T11:19:51.471 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.492+0000 7f23f6ffd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:51.471 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.492+0000 7f23f6ffd640 1 -- 172.21.15.47:0/2834540928 >> v1:172.21.15.47:6789/0 conn(0x7f24100a9ac0 legacy=0x7f241006f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:51.472 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.492+0000 7f23f6ffd640 1 -- 172.21.15.47:0/2834540928 shutdown_connections 2024-03-29T11:19:51.472 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.492+0000 7f23f6ffd640 1 -- 172.21.15.47:0/2834540928 >> 172.21.15.47:0/2834540928 conn(0x7f24100b35e0 msgr2=0x7f241013b100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:51.472 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.492+0000 7f23f6ffd640 1 -- 172.21.15.47:0/2834540928 shutdown_connections 2024-03-29T11:19:51.472 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.492+0000 7f23f6ffd640 1 -- 172.21.15.47:0/2834540928 wait complete. 2024-03-29T11:19:51.472 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:51.492+0000 7f23f6ffd640 1 librados: shutdown 2024-03-29T11:19:51.486 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:58.294 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:58.445 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.466+0000 7fc9a8c46640 10 monclient: get_monmap_and_config 2024-03-29T11:19:58.445 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.466+0000 7fc9a8c46640 10 monclient: build_initial_monmap 2024-03-29T11:19:58.445 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.466+0000 7fc9a8c46640 10 monclient: monmap: 2024-03-29T11:19:58.445 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:19:58.445 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:19:58.445 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:19:58.467164+0000 2024-03-29T11:19:58.445 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:19:58.467164+0000 2024-03-29T11:19:58.445 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:58.445 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:19:58.445 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:19:58.445 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:19:58.446 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:19:58.446 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:19:58.446 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.467+0000 7fc9a8c46640 1 Processor -- start 2024-03-29T11:19:58.446 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.468+0000 7fc9a8c46640 1 -- start start 2024-03-29T11:19:58.446 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.468+0000 7fc9a8c46640 10 monclient: init 2024-03-29T11:19:58.449 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.471+0000 7fc9a8c46640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:58.449 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.471+0000 7fc9a8c46640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:19:58.449 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.471+0000 7fc9a8c46640 10 monclient(hunting): picked mon.noname-b con 0x7fc99c034850 addr v1:172.21.15.47:6790/0 2024-03-29T11:19:58.449 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.471+0000 7fc9a8c46640 10 monclient(hunting): picked mon.noname-a con 0x7fc99c0c9550 addr v1:172.21.15.47:6789/0 2024-03-29T11:19:58.449 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.471+0000 7fc9a8c46640 10 monclient(hunting): picked mon.noname-c con 0x7fc99c02aac0 addr v1:172.21.15.121:6789/0 2024-03-29T11:19:58.449 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.471+0000 7fc9a8c46640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc99c02c440 con 0x7fc99c0c9550 2024-03-29T11:19:58.449 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.471+0000 7fc9a8c46640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc99c02aed0 con 0x7fc99c02aac0 2024-03-29T11:19:58.450 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.471+0000 7fc9a8c46640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc99c02b190 con 0x7fc99c034850 2024-03-29T11:19:58.450 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.471+0000 7fc9a8c46640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:58.450 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.471+0000 7fc9a8c46640 10 monclient(hunting): authenticate will time out at 1241.659152s 2024-03-29T11:19:58.450 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.473+0000 7fc9a37fe640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fc99c034850 0x7fc99c0c8e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:49732/0 (socket says 172.21.15.47:49732) 2024-03-29T11:19:58.450 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.473+0000 7fc9a37fe640 1 -- 172.21.15.47:0/182602595 learned_addr learned my addr 172.21.15.47:0/182602595 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:19:58.450 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.473+0000 7fc9a27fc640 1 -- 172.21.15.47:0/182602595 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2720965192 0 0) 0x7fc99c02b190 con 0x7fc99c034850 2024-03-29T11:19:58.451 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.473+0000 7fc9a27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:58.451 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.473+0000 7fc9a27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:58.451 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.473+0000 7fc9a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:58.451 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.473+0000 7fc9a27fc640 10 monclient(hunting): my global_id is 4514 2024-03-29T11:19:58.451 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.473+0000 7fc9a27fc640 1 -- 172.21.15.47:0/182602595 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc994003520 con 0x7fc99c034850 2024-03-29T11:19:58.454 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.475+0000 7fc9a27fc640 1 -- 172.21.15.47:0/182602595 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1921807433 0 0) 0x7fc994003520 con 0x7fc99c034850 2024-03-29T11:19:58.454 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.475+0000 7fc9a27fc640 1 -- 172.21.15.47:0/182602595 >> v1:172.21.15.121:6789/0 conn(0x7fc99c02aac0 legacy=0x7fc99c0cc5b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:19:58.454 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.475+0000 7fc9a27fc640 1 -- 172.21.15.47:0/182602595 >> v1:172.21.15.47:6789/0 conn(0x7fc99c0c9550 legacy=0x7fc99c0cbea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:19:58.454 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.476+0000 7fc9a27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:19:58.455 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.476+0000 7fc9a27fc640 1 monclient: found mon.noname-b 2024-03-29T11:19:58.455 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.476+0000 7fc9a27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:58.455 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.476+0000 7fc9a27fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-29T11:19:58.455 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.476+0000 7fc9a27fc640 1 -- 172.21.15.47:0/182602595 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc99c033060 con 0x7fc99c034850 2024-03-29T11:19:58.455 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.476+0000 7fc9a27fc640 10 monclient: _finish_auth 0 2024-03-29T11:19:58.455 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.476+0000 7fc9a27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:19:58.455 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.476+0000 7fc9a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:58.455 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.476+0000 7fc9a8c46640 5 monclient: authenticate success, global_id 4514 2024-03-29T11:19:58.455 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.476+0000 7fc9a8c46640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:58.455 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.476+0000 7fc9a27fc640 1 -- 172.21.15.47:0/182602595 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fc998001690 con 0x7fc99c034850 2024-03-29T11:19:58.455 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.476+0000 7fc9a27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:58.455 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.476+0000 7fc9a27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:58.455 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.476+0000 7fc9a27fc640 10 monclient: dump: 2024-03-29T11:19:58.455 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:19:58.456 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:19:58.456 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:58.456 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:58.456 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:58.456 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:19:58.456 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:19:58.456 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:19:58.456 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:19:58.456 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:19:58.456 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.476+0000 7fc9a8c46640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:58.456 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.477+0000 7fc9a27fc640 1 -- 172.21.15.47:0/182602595 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc998001920 con 0x7fc99c034850 2024-03-29T11:19:58.456 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.477+0000 7fc9a27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:58.456 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.477+0000 7fc9a8c46640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:58.456 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.477+0000 7fc9a8c46640 4 set_mon_vals no callback set 2024-03-29T11:19:58.456 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.478+0000 7fc9a27fc640 1 -- 172.21.15.47:0/182602595 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fc998001cb0 con 0x7fc99c034850 2024-03-29T11:19:58.457 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.478+0000 7fc9a27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:58.457 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.478+0000 7fc9a27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:58.457 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.478+0000 7fc9a27fc640 10 monclient: dump: 2024-03-29T11:19:58.457 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:19:58.457 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:19:58.457 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:58.457 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:58.457 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:58.457 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:19:58.457 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:19:58.457 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:19:58.457 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:19:58.457 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:19:58.459 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.478+0000 7fc9a8c46640 10 monclient: shutdown 2024-03-29T11:19:58.459 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.478+0000 7fc9a8c46640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:58.459 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.478+0000 7fc9a8c46640 1 -- 172.21.15.47:0/182602595 >> v1:172.21.15.47:6790/0 conn(0x7fc99c034850 legacy=0x7fc99c0c8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:58.459 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.478+0000 7fc9a8c46640 1 -- 172.21.15.47:0/182602595 shutdown_connections 2024-03-29T11:19:58.459 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.478+0000 7fc9a8c46640 1 -- 172.21.15.47:0/182602595 >> 172.21.15.47:0/182602595 conn(0x7fc99c0058f0 msgr2=0x7fc99c005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:58.460 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.479+0000 7fc9a8c46640 1 -- 172.21.15.47:0/182602595 shutdown_connections 2024-03-29T11:19:58.460 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.479+0000 7fc9a8c46640 1 -- 172.21.15.47:0/182602595 wait complete. 2024-03-29T11:19:58.460 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.480+0000 7fc9a8c46640 10 monclient: build_initial_monmap 2024-03-29T11:19:58.460 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.480+0000 7fc9a8c46640 10 monclient: monmap: 2024-03-29T11:19:58.460 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:19:58.460 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:58.460 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:19:58.460 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:19:58.460 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:58.460 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:19:58.460 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:19:58.460 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:19:58.460 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:19:58.460 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:19:58.461 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.480+0000 7fc9a8c46640 1 librados: starting msgr at 2024-03-29T11:19:58.461 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.480+0000 7fc9a8c46640 1 librados: starting objecter 2024-03-29T11:19:58.461 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.480+0000 7fc9a8c46640 1 Processor -- start 2024-03-29T11:19:58.461 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.481+0000 7fc9a8c46640 1 -- start start 2024-03-29T11:19:58.461 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.481+0000 7fc9a8c46640 1 librados: setting wanted keys 2024-03-29T11:19:58.461 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.481+0000 7fc9a8c46640 1 librados: calling monclient init 2024-03-29T11:19:58.461 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.481+0000 7fc9a8c46640 10 monclient: init 2024-03-29T11:19:58.461 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.481+0000 7fc9a8c46640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:58.461 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.481+0000 7fc9a8c46640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:19:58.461 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.481+0000 7fc9a8c46640 10 monclient(hunting): picked mon.noname-c con 0x7fc99c02aac0 addr v1:172.21.15.47:6790/0 2024-03-29T11:19:58.461 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.481+0000 7fc9a8c46640 10 monclient(hunting): picked mon.noname-a con 0x7fc99c034850 addr v1:172.21.15.47:6789/0 2024-03-29T11:19:58.461 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.481+0000 7fc9a8c46640 10 monclient(hunting): picked mon.noname-b con 0x7fc99c0c9550 addr v1:172.21.15.121:6789/0 2024-03-29T11:19:58.461 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.481+0000 7fc9a8c46640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc99c0bfc20 con 0x7fc99c034850 2024-03-29T11:19:58.461 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.481+0000 7fc9a8c46640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc99c0bf8d0 con 0x7fc99c0c9550 2024-03-29T11:19:58.461 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.481+0000 7fc9a8c46640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc99c0c07c0 con 0x7fc99c02aac0 2024-03-29T11:19:58.462 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.481+0000 7fc9a8c46640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:58.462 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.481+0000 7fc9a8c46640 10 monclient(hunting): authenticate will time out at 1241.669863s 2024-03-29T11:19:58.462 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.483+0000 7fc9a37fe640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fc99c02aac0 0x7fc99c0c00b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:55398/0 (socket says 172.21.15.47:55398) 2024-03-29T11:19:58.462 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.483+0000 7fc9a37fe640 1 -- 172.21.15.47:0/728827837 learned_addr learned my addr 172.21.15.47:0/728827837 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:19:58.462 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.483+0000 7fc98ffff640 10 client.?.objecter ms_handle_connect 0x7fc99c02aac0 2024-03-29T11:19:58.462 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.483+0000 7fc98ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:58.463 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.483+0000 7fc98ffff640 1 -- 172.21.15.47:0/728827837 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 751736642 0 0) 0x7fc99c0c07c0 con 0x7fc99c02aac0 2024-03-29T11:19:58.463 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.483+0000 7fc98ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:58.463 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.483+0000 7fc98ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:58.463 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.483+0000 7fc98ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:58.463 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.483+0000 7fc98ffff640 10 monclient(hunting): my global_id is 4517 2024-03-29T11:19:58.463 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.483+0000 7fc98ffff640 1 -- 172.21.15.47:0/728827837 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc988004eb0 con 0x7fc99c02aac0 2024-03-29T11:19:58.463 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.484+0000 7fc98ffff640 1 -- 172.21.15.47:0/728827837 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4147778869 0 0) 0x7fc988004eb0 con 0x7fc99c02aac0 2024-03-29T11:19:58.463 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.484+0000 7fc98ffff640 1 -- 172.21.15.47:0/728827837 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc99c0c07c0 con 0x7fc99c02aac0 2024-03-29T11:19:58.463 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.484+0000 7fc98ffff640 1 -- 172.21.15.47:0/728827837 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fc998001610 con 0x7fc99c02aac0 2024-03-29T11:19:58.463 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.484+0000 7fc98ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:58.463 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.484+0000 7fc98ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:58.463 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.484+0000 7fc98ffff640 10 monclient(hunting): dump: 2024-03-29T11:19:58.463 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:19:58.464 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:19:58.464 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:58.464 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:58.464 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:58.464 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:19:58.464 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:19:58.464 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:19:58.464 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:19:58.464 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:19:58.464 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.484+0000 7fc98ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:58.464 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.484+0000 7fc9a37fe640 1 -- 172.21.15.47:0/728827837 >> v1:172.21.15.47:6790/0 conn(0x7fc99c02aac0 legacy=0x7fc99c0c00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-29T11:19:58.464 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.484+0000 7fc9a37fe640 1 -- 172.21.15.47:0/728827837 >> v1:172.21.15.47:6790/0 conn(0x7fc99c02aac0 legacy=0x7fc99c0c00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:19:58.464 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.484+0000 7fc9a37fe640 1 --1- 172.21.15.47:0/728827837 >> v1:172.21.15.47:6790/0 conn(0x7fc99c02aac0 0x7fc99c0c00b0 :-1 s=OPENED pgs=716615 cs=1 l=1).handle_message read tag failed 2024-03-29T11:19:58.464 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.484+0000 7fc9a37fe640 1 --1- 172.21.15.47:0/728827837 >> v1:172.21.15.47:6790/0 conn(0x7fc99c02aac0 0x7fc99c0c00b0 :-1 s=OPENED pgs=716615 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:19:58.464 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.484+0000 7fc98ffff640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.47:6790/0 2024-03-29T11:19:58.464 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.485+0000 7fc98ffff640 10 client.?.objecter ms_handle_connect 0x7fc99c0c9550 2024-03-29T11:19:58.465 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.485+0000 7fc98ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:58.467 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.486+0000 7fc98ffff640 1 -- 172.21.15.47:0/728827837 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1042864135 0 0) 0x7fc99c0bf8d0 con 0x7fc99c0c9550 2024-03-29T11:19:58.467 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.486+0000 7fc98ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:58.467 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.486+0000 7fc98ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:58.467 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.486+0000 7fc98ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:58.467 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.486+0000 7fc98ffff640 10 monclient(hunting): my global_id is 4462 2024-03-29T11:19:58.467 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.486+0000 7fc98ffff640 1 -- 172.21.15.47:0/728827837 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc988004eb0 con 0x7fc99c0c9550 2024-03-29T11:19:58.468 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.486+0000 7fc98ffff640 1 -- 172.21.15.47:0/728827837 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 346805196 0 0) 0x7fc988004eb0 con 0x7fc99c0c9550 2024-03-29T11:19:58.468 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.487+0000 7fc98ffff640 1 -- 172.21.15.47:0/728827837 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc99c0bf8d0 con 0x7fc99c0c9550 2024-03-29T11:19:58.468 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.487+0000 7fc98ffff640 1 -- 172.21.15.47:0/728827837 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fc9a404fdd0 con 0x7fc99c0c9550 2024-03-29T11:19:58.468 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.487+0000 7fc98ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:58.468 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.487+0000 7fc98ffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:58.468 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.487+0000 7fc98ffff640 10 monclient(hunting): dump: 2024-03-29T11:19:58.468 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:19:58.468 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:19:58.468 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:58.468 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:58.468 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:58.468 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:19:58.469 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:19:58.469 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:19:58.469 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:19:58.469 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:19:58.469 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.487+0000 7fc98ffff640 1 -- 172.21.15.47:0/728827837 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 439849433 0 0) 0x7fc99c0bf8d0 con 0x7fc99c0c9550 2024-03-29T11:19:58.469 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.487+0000 7fc98ffff640 1 -- 172.21.15.47:0/728827837 >> v1:172.21.15.47:6790/0 conn(0x7fc99c02aac0 legacy=0x7fc99c0c00b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:19:58.469 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.487+0000 7fc98ffff640 1 -- 172.21.15.47:0/728827837 >> v1:172.21.15.47:6789/0 conn(0x7fc99c034850 legacy=0x7fc99c17f7f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:19:58.469 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.488+0000 7fc98ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:19:58.469 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.488+0000 7fc98ffff640 1 monclient: found mon.b 2024-03-29T11:19:58.469 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.488+0000 7fc98ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:58.469 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.488+0000 7fc98ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:19:58.469 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.488+0000 7fc98ffff640 1 -- 172.21.15.47:0/728827837 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc99c180610 con 0x7fc99c0c9550 2024-03-29T11:19:58.470 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.489+0000 7fc98ffff640 10 monclient: _finish_auth 0 2024-03-29T11:19:58.470 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.489+0000 7fc98ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:19:58.470 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.489+0000 7fc98ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:58.470 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.489+0000 7fc9a8c46640 5 monclient: authenticate success, global_id 4462 2024-03-29T11:19:58.470 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.489+0000 7fc9a8c46640 10 monclient: _renew_subs 2024-03-29T11:19:58.470 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.489+0000 7fc9a8c46640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:19:58.470 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.489+0000 7fc9a8c46640 1 -- 172.21.15.47:0/728827837 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc99c186da0 con 0x7fc99c0c9550 2024-03-29T11:19:58.470 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.489+0000 7fc9a8c46640 10 client.4462.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:58.470 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.489+0000 7fc9a8c46640 10 monclient: _renew_subs 2024-03-29T11:19:58.470 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.489+0000 7fc9a8c46640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:19:58.470 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.489+0000 7fc9a8c46640 1 -- 172.21.15.47:0/728827837 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc99c0bf5f0 con 0x7fc99c0c9550 2024-03-29T11:19:58.470 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.489+0000 7fc9a8c46640 1 librados: init done 2024-03-29T11:19:58.470 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.489+0000 7fc98ffff640 1 -- 172.21.15.47:0/728827837 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc9a404f400 con 0x7fc99c0c9550 2024-03-29T11:19:58.473 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.489+0000 7fc98ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:58.473 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.489+0000 7fc9a1ffb640 4 set_mon_vals no callback set 2024-03-29T11:19:58.473 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.490+0000 7fc9a8c46640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:58.473 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.490+0000 7fc98ffff640 1 -- 172.21.15.47:0/728827837 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fc9a4075f10 con 0x7fc99c0c9550 2024-03-29T11:19:58.473 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.490+0000 7fc98ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:58.473 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.490+0000 7fc98ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:58.473 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.490+0000 7fc98ffff640 10 monclient: dump: 2024-03-29T11:19:58.473 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:19:58.473 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:19:58.473 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:58.473 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:19:58.473 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:58.473 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:19:58.474 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:19:58.474 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:19:58.474 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:19:58.474 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:19:58.474 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.490+0000 7fc9a8c46640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:58.474 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.490+0000 7fc9a8c46640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:19:58.474 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.490+0000 7fc9a8c46640 1 -- 172.21.15.47:0/728827837 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc984005080 con 0x7fc99c0c9550 2024-03-29T11:19:58.474 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.492+0000 7fc98ffff640 1 -- 172.21.15.47:0/728827837 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7fc9a4067bf0 con 0x7fc99c0c9550 2024-03-29T11:19:58.474 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.492+0000 7fc98ffff640 1 -- 172.21.15.47:0/728827837 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7fc9a40ed110 con 0x7fc99c0c9550 2024-03-29T11:19:58.474 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.492+0000 7fc98ffff640 10 client.4462.objecter ms_dispatch 0x7fc9a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:58.474 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.492+0000 7fc98ffff640 3 client.4462.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:58.474 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.492+0000 7fc98ffff640 3 client.4462.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:58.474 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.492+0000 7fc98ffff640 20 client.4462.objecter dump_active .. 0 homeless 2024-03-29T11:19:58.474 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.493+0000 7fc9a37fe640 10 monclient: get_auth_request con 0x7fc98806de60 auth_method 0 2024-03-29T11:19:58.475 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.494+0000 7fc98ffff640 10 client.4462.objecter ms_handle_connect 0x7fc98806de60 2024-03-29T11:19:58.477 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.500+0000 7fc98ffff640 1 -- 172.21.15.47:0/728827837 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc9a40b89f0 con 0x7fc99c0c9550 2024-03-29T11:19:58.477 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.500+0000 7fc98ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:58.477 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.500+0000 7fc98ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:58.804 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.827+0000 7fc9a8c46640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:58.804 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.827+0000 7fc9a8c46640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:58.804 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.827+0000 7fc9a8c46640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:19:58.804 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.827+0000 7fc9a8c46640 1 -- 172.21.15.47:0/728827837 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc984005330 con 0x7fc99c0c9550 2024-03-29T11:19:58.805 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.828+0000 7fc98ffff640 1 -- 172.21.15.47:0/728827837 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7fc9a40ba920 con 0x7fc99c0c9550 2024-03-29T11:19:58.805 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.828+0000 7fc98ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:58.805 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.828+0000 7fc98ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:58.805 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:19:58.805 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:58.808 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.832+0000 7fc9a8c46640 10 librados: watch_flush enter 2024-03-29T11:19:58.809 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.832+0000 7fc9a8c46640 10 librados: watch_flush exit 2024-03-29T11:19:58.809 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.832+0000 7fc9a8c46640 20 client.4462.objecter shutdown clearing up homeless session... 2024-03-29T11:19:58.809 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.832+0000 7fc9a8c46640 10 client.4462.objecter successfully canceled tick 2024-03-29T11:19:58.810 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.833+0000 7fc9a8c46640 1 -- 172.21.15.47:0/728827837 >> v1:172.21.15.121:6816/1964117370 conn(0x7fc98806de60 legacy=0x7fc988070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:58.810 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.833+0000 7fc9a8c46640 10 monclient: shutdown 2024-03-29T11:19:58.810 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.833+0000 7fc9a8c46640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:58.810 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.833+0000 7fc9a8c46640 1 -- 172.21.15.47:0/728827837 >> v1:172.21.15.121:6789/0 conn(0x7fc99c0c9550 legacy=0x7fc99c17ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:58.810 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.834+0000 7fc9a8c46640 1 -- 172.21.15.47:0/728827837 shutdown_connections 2024-03-29T11:19:58.810 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.834+0000 7fc9a8c46640 1 -- 172.21.15.47:0/728827837 >> 172.21.15.47:0/728827837 conn(0x7fc99c0058f0 msgr2=0x7fc99c0043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:58.812 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.835+0000 7fc9a8c46640 1 -- 172.21.15.47:0/728827837 shutdown_connections 2024-03-29T11:19:58.812 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.835+0000 7fc9a8c46640 1 -- 172.21.15.47:0/728827837 wait complete. 2024-03-29T11:19:58.814 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:19:58.838+0000 7fc9a8c46640 1 librados: shutdown 2024-03-29T11:19:58.833 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:05.639 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:05.793 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.815+0000 7fe2198b3640 10 monclient: get_monmap_and_config 2024-03-29T11:20:05.793 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.815+0000 7fe2198b3640 10 monclient: build_initial_monmap 2024-03-29T11:20:05.793 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.815+0000 7fe2198b3640 10 monclient: monmap: 2024-03-29T11:20:05.793 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:20:05.793 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:20:05.793 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:20:05.816282+0000 2024-03-29T11:20:05.793 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:20:05.816282+0000 2024-03-29T11:20:05.793 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:05.793 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:20:05.794 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:20:05.794 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:20:05.794 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:20:05.794 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:20:05.794 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.817+0000 7fe2198b3640 1 Processor -- start 2024-03-29T11:20:05.794 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.817+0000 7fe2198b3640 1 -- start start 2024-03-29T11:20:05.795 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.817+0000 7fe2198b3640 10 monclient: init 2024-03-29T11:20:05.795 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.818+0000 7fe2198b3640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:05.795 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.818+0000 7fe2198b3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:20:05.795 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.818+0000 7fe2198b3640 10 monclient(hunting): picked mon.noname-c con 0x7fe2140a9ac0 addr v1:172.21.15.121:6789/0 2024-03-29T11:20:05.795 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.818+0000 7fe2198b3640 10 monclient(hunting): picked mon.noname-b con 0x7fe21413ee90 addr v1:172.21.15.47:6790/0 2024-03-29T11:20:05.796 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.818+0000 7fe2198b3640 10 monclient(hunting): picked mon.noname-a con 0x7fe2140af900 addr v1:172.21.15.47:6789/0 2024-03-29T11:20:05.796 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.818+0000 7fe2198b3640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2140afd10 con 0x7fe2140af900 2024-03-29T11:20:05.796 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.818+0000 7fe2198b3640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2140af6f0 con 0x7fe2140a9ac0 2024-03-29T11:20:05.796 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.818+0000 7fe2198b3640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2140aac80 con 0x7fe21413ee90 2024-03-29T11:20:05.796 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.818+0000 7fe2198b3640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:05.796 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.818+0000 7fe2198b3640 10 monclient(hunting): authenticate will time out at 1249.006113s 2024-03-29T11:20:05.796 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.818+0000 7fe212ffd640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fe2140a9ac0 0x7fe2140adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.47:56040/0 (socket says 172.21.15.47:56040) 2024-03-29T11:20:05.796 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.818+0000 7fe212ffd640 1 -- 172.21.15.47:0/1030754542 learned_addr learned my addr 172.21.15.47:0/1030754542 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:20:05.796 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.818+0000 7fe211ffb640 1 -- 172.21.15.47:0/1030754542 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 119483032 0 0) 0x7fe2140af6f0 con 0x7fe2140a9ac0 2024-03-29T11:20:05.796 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.818+0000 7fe211ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:20:05.796 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.818+0000 7fe211ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:05.796 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.818+0000 7fe211ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:05.796 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.818+0000 7fe211ffb640 10 monclient(hunting): my global_id is 4465 2024-03-29T11:20:05.796 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.819+0000 7fe211ffb640 1 -- 172.21.15.47:0/1030754542 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe200003520 con 0x7fe2140a9ac0 2024-03-29T11:20:05.797 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.819+0000 7fe211ffb640 1 -- 172.21.15.47:0/1030754542 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1464668742 0 0) 0x7fe200003520 con 0x7fe2140a9ac0 2024-03-29T11:20:05.797 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.819+0000 7fe211ffb640 1 -- 172.21.15.47:0/1030754542 >> v1:172.21.15.47:6790/0 conn(0x7fe21413ee90 legacy=0x7fe21413b5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:20:05.797 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.819+0000 7fe211ffb640 1 -- 172.21.15.47:0/1030754542 >> v1:172.21.15.47:6789/0 conn(0x7fe2140af900 legacy=0x7fe214149b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:20:05.799 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.821+0000 7fe211ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:20:05.799 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.821+0000 7fe211ffb640 1 monclient: found mon.noname-c 2024-03-29T11:20:05.799 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.821+0000 7fe211ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:05.799 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.821+0000 7fe211ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:20:05.799 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.821+0000 7fe211ffb640 1 -- 172.21.15.47:0/1030754542 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2140aaf60 con 0x7fe2140a9ac0 2024-03-29T11:20:05.799 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.821+0000 7fe211ffb640 10 monclient: _finish_auth 0 2024-03-29T11:20:05.799 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.821+0000 7fe211ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:20:05.799 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.821+0000 7fe211ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:05.799 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.821+0000 7fe211ffb640 1 -- 172.21.15.47:0/1030754542 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fe204001a10 con 0x7fe2140a9ac0 2024-03-29T11:20:05.799 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.821+0000 7fe211ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:05.800 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.821+0000 7fe211ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:05.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.821+0000 7fe211ffb640 10 monclient: dump: 2024-03-29T11:20:05.802 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:20:05.802 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:20:05.802 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:05.802 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:05.802 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:05.802 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:20:05.802 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:20:05.802 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:20:05.802 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:20:05.802 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:20:05.802 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.821+0000 7fe2198b3640 5 monclient: authenticate success, global_id 4465 2024-03-29T11:20:05.802 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.821+0000 7fe2198b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:05.802 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.821+0000 7fe211ffb640 1 -- 172.21.15.47:0/1030754542 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe204001ca0 con 0x7fe2140a9ac0 2024-03-29T11:20:05.802 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.821+0000 7fe211ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:05.803 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.822+0000 7fe2198b3640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:05.803 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.822+0000 7fe2198b3640 4 set_mon_vals no callback set 2024-03-29T11:20:05.803 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.823+0000 7fe2198b3640 10 monclient: shutdown 2024-03-29T11:20:05.803 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.823+0000 7fe2198b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:05.803 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.823+0000 7fe2198b3640 1 -- 172.21.15.47:0/1030754542 >> v1:172.21.15.121:6789/0 conn(0x7fe2140a9ac0 legacy=0x7fe2140adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:05.803 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.823+0000 7fe2198b3640 1 -- 172.21.15.47:0/1030754542 shutdown_connections 2024-03-29T11:20:05.803 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.823+0000 7fe2198b3640 1 -- 172.21.15.47:0/1030754542 >> 172.21.15.47:0/1030754542 conn(0x7fe2140b35e0 msgr2=0x7fe2140b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:05.803 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.823+0000 7fe2198b3640 1 -- 172.21.15.47:0/1030754542 shutdown_connections 2024-03-29T11:20:05.803 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.824+0000 7fe2198b3640 1 -- 172.21.15.47:0/1030754542 wait complete. 2024-03-29T11:20:05.803 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.825+0000 7fe2198b3640 10 monclient: build_initial_monmap 2024-03-29T11:20:05.803 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.825+0000 7fe2198b3640 10 monclient: monmap: 2024-03-29T11:20:05.803 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:20:05.803 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:05.803 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:20:05.803 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:20:05.804 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:05.804 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:20:05.804 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:20:05.804 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:20:05.804 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:20:05.804 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:20:05.806 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.827+0000 7fe2198b3640 1 librados: starting msgr at 2024-03-29T11:20:05.806 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.827+0000 7fe2198b3640 1 librados: starting objecter 2024-03-29T11:20:05.806 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.827+0000 7fe2198b3640 1 Processor -- start 2024-03-29T11:20:05.806 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.827+0000 7fe2198b3640 1 -- start start 2024-03-29T11:20:05.806 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.827+0000 7fe2198b3640 1 librados: setting wanted keys 2024-03-29T11:20:05.806 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.827+0000 7fe2198b3640 1 librados: calling monclient init 2024-03-29T11:20:05.806 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.827+0000 7fe2198b3640 10 monclient: init 2024-03-29T11:20:05.807 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.828+0000 7fe2198b3640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:05.807 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.828+0000 7fe2198b3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:20:05.807 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.828+0000 7fe2198b3640 10 monclient(hunting): picked mon.noname-a con 0x7fe2140af900 addr v1:172.21.15.47:6789/0 2024-03-29T11:20:05.807 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.828+0000 7fe2198b3640 10 monclient(hunting): picked mon.noname-b con 0x7fe21413ee90 addr v1:172.21.15.121:6789/0 2024-03-29T11:20:05.807 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.828+0000 7fe2198b3640 10 monclient(hunting): picked mon.noname-c con 0x7fe21413c7d0 addr v1:172.21.15.47:6790/0 2024-03-29T11:20:05.807 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.828+0000 7fe2198b3640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe21413df00 con 0x7fe2140af900 2024-03-29T11:20:05.807 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.828+0000 7fe2198b3640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe21413f6f0 con 0x7fe21413ee90 2024-03-29T11:20:05.807 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.828+0000 7fe2198b3640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2141def10 con 0x7fe21413c7d0 2024-03-29T11:20:05.807 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.828+0000 7fe2198b3640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:05.807 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.828+0000 7fe2198b3640 10 monclient(hunting): authenticate will time out at 1249.016453s 2024-03-29T11:20:05.807 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.829+0000 7fe2127fc640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fe21413ee90 0x7fe214131f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.47:57418/0 (socket says 172.21.15.47:57418) 2024-03-29T11:20:05.807 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.829+0000 7fe2127fc640 1 -- 172.21.15.47:0/2268383951 learned_addr learned my addr 172.21.15.47:0/2268383951 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:20:05.808 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.829+0000 7fe20bfff640 10 client.?.objecter ms_handle_connect 0x7fe21413ee90 2024-03-29T11:20:05.808 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.829+0000 7fe20bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:05.808 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.830+0000 7fe20bfff640 1 -- 172.21.15.47:0/2268383951 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1654366872 0 0) 0x7fe21413f6f0 con 0x7fe21413ee90 2024-03-29T11:20:05.809 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.830+0000 7fe20bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:05.809 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.830+0000 7fe20bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:05.809 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.830+0000 7fe20bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:05.809 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.830+0000 7fe20bfff640 10 monclient(hunting): my global_id is 4468 2024-03-29T11:20:05.809 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.830+0000 7fe20bfff640 1 -- 172.21.15.47:0/2268383951 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1f4004eb0 con 0x7fe21413ee90 2024-03-29T11:20:05.809 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.830+0000 7fe20bfff640 1 -- 172.21.15.47:0/2268383951 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2454640860 0 0) 0x7fe1f4004eb0 con 0x7fe21413ee90 2024-03-29T11:20:05.809 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.830+0000 7fe20bfff640 1 -- 172.21.15.47:0/2268383951 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe21413f6f0 con 0x7fe21413ee90 2024-03-29T11:20:05.809 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.831+0000 7fe20bfff640 1 -- 172.21.15.47:0/2268383951 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fe20c001870 con 0x7fe21413ee90 2024-03-29T11:20:05.809 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.831+0000 7fe20bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:05.809 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.831+0000 7fe20bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:05.809 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.831+0000 7fe20bfff640 10 monclient(hunting): dump: 2024-03-29T11:20:05.809 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:20:05.809 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:20:05.809 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:05.810 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:05.810 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:05.810 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:20:05.810 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:20:05.810 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:20:05.810 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:20:05.810 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:20:05.810 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.831+0000 7fe20bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:05.810 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.831+0000 7fe20bfff640 1 -- 172.21.15.47:0/2268383951 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1180598726 0 0) 0x7fe21413f6f0 con 0x7fe21413ee90 2024-03-29T11:20:05.810 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.831+0000 7fe20bfff640 1 -- 172.21.15.47:0/2268383951 >> v1:172.21.15.47:6790/0 conn(0x7fe21413c7d0 legacy=0x7fe2141e69f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:20:05.812 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.833+0000 7fe20bfff640 1 -- 172.21.15.47:0/2268383951 >> v1:172.21.15.47:6789/0 conn(0x7fe2140af900 legacy=0x7fe21413c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:05.812 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.833+0000 7fe20bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:20:05.812 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.833+0000 7fe20bfff640 1 monclient: found mon.b 2024-03-29T11:20:05.813 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.833+0000 7fe20bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:05.813 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.833+0000 7fe20bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:20:05.813 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.833+0000 7fe20bfff640 1 -- 172.21.15.47:0/2268383951 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2141deb50 con 0x7fe21413ee90 2024-03-29T11:20:05.813 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.833+0000 7fe20bfff640 10 monclient: _finish_auth 0 2024-03-29T11:20:05.813 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.833+0000 7fe20bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:20:05.813 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.833+0000 7fe20bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:05.813 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.833+0000 7fe2198b3640 5 monclient: authenticate success, global_id 4468 2024-03-29T11:20:05.813 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.833+0000 7fe2198b3640 10 monclient: _renew_subs 2024-03-29T11:20:05.813 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.833+0000 7fe2198b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:20:05.813 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.833+0000 7fe2198b3640 1 -- 172.21.15.47:0/2268383951 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2141df430 con 0x7fe21413ee90 2024-03-29T11:20:05.813 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.833+0000 7fe2198b3640 10 client.4468.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:05.813 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.833+0000 7fe2198b3640 10 monclient: _renew_subs 2024-03-29T11:20:05.813 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.833+0000 7fe2198b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:20:05.813 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.833+0000 7fe2198b3640 1 -- 172.21.15.47:0/2268383951 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe214132680 con 0x7fe21413ee90 2024-03-29T11:20:05.813 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.833+0000 7fe2198b3640 1 librados: init done 2024-03-29T11:20:05.813 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.833+0000 7fe20bfff640 10 client.4468.objecter ms_handle_connect 0x7fe2140af900 2024-03-29T11:20:05.814 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.833+0000 7fe20bfff640 10 client.4468.objecter resend_mon_ops 2024-03-29T11:20:05.814 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.833+0000 7fe20bfff640 1 -- 172.21.15.47:0/2268383951 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe20c0019d0 con 0x7fe21413ee90 2024-03-29T11:20:05.814 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.833+0000 7fe20bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:05.814 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.833+0000 7fe20bfff640 1 -- 172.21.15.47:0/2268383951 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fe20c004b00 con 0x7fe21413ee90 2024-03-29T11:20:05.814 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.833+0000 7fe20bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:05.814 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.833+0000 7fe20bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:05.814 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.833+0000 7fe20bfff640 10 monclient: dump: 2024-03-29T11:20:05.814 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:20:05.814 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:20:05.814 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:05.814 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:05.814 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:05.814 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:20:05.815 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:20:05.815 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:20:05.815 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:20:05.815 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:20:05.815 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.835+0000 7fe210ff9640 4 set_mon_vals no callback set 2024-03-29T11:20:05.815 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.835+0000 7fe2198b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:05.817 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.836+0000 7fe20bfff640 1 -- 172.21.15.47:0/2268383951 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7fe20c01aad0 con 0x7fe21413ee90 2024-03-29T11:20:05.817 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.836+0000 7fe2198b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:05.817 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.836+0000 7fe2198b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:20:05.817 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.836+0000 7fe2198b3640 1 -- 172.21.15.47:0/2268383951 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1f8005080 con 0x7fe21413ee90 2024-03-29T11:20:05.817 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.837+0000 7fe20bfff640 1 -- 172.21.15.47:0/2268383951 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7fe20c084b10 con 0x7fe21413ee90 2024-03-29T11:20:05.817 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.837+0000 7fe20bfff640 10 client.4468.objecter ms_dispatch 0x7fe214000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:05.818 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.837+0000 7fe20bfff640 3 client.4468.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:05.818 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.837+0000 7fe20bfff640 3 client.4468.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:05.818 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.838+0000 7fe20bfff640 20 client.4468.objecter dump_active .. 0 homeless 2024-03-29T11:20:05.818 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.838+0000 7fe212ffd640 10 monclient: get_auth_request con 0x7fe1f406dbe0 auth_method 0 2024-03-29T11:20:05.818 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.839+0000 7fe20bfff640 10 client.4468.objecter ms_handle_connect 0x7fe1f406dbe0 2024-03-29T11:20:05.821 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.843+0000 7fe20bfff640 1 -- 172.21.15.47:0/2268383951 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe20c0503f0 con 0x7fe21413ee90 2024-03-29T11:20:05.821 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.843+0000 7fe20bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:05.821 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:05.843+0000 7fe20bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:06.223 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:06.247+0000 7fe2198b3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:06.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:06.247+0000 7fe2198b3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:06.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:06.247+0000 7fe2198b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:20:06.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:06.247+0000 7fe2198b3640 1 -- 172.21.15.47:0/2268383951 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe1f8005330 con 0x7fe21413ee90 2024-03-29T11:20:06.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:06.248+0000 7fe20bfff640 1 -- 172.21.15.47:0/2268383951 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7fe20c052320 con 0x7fe21413ee90 2024-03-29T11:20:06.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:06.248+0000 7fe20bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:06.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:06.248+0000 7fe20bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:06.227 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:20:06.227 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:06.230 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:06.253+0000 7fe2198b3640 10 librados: watch_flush enter 2024-03-29T11:20:06.231 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:06.254+0000 7fe2198b3640 10 librados: watch_flush exit 2024-03-29T11:20:06.231 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:06.254+0000 7fe2198b3640 20 client.4468.objecter shutdown clearing up homeless session... 2024-03-29T11:20:06.231 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:06.254+0000 7fe2198b3640 10 client.4468.objecter successfully canceled tick 2024-03-29T11:20:06.231 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:06.254+0000 7fe2198b3640 1 -- 172.21.15.47:0/2268383951 >> v1:172.21.15.121:6816/1964117370 conn(0x7fe1f406dbe0 legacy=0x7fe1f40700a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:06.231 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:06.254+0000 7fe2198b3640 10 monclient: shutdown 2024-03-29T11:20:06.231 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:06.254+0000 7fe2198b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:06.231 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:06.254+0000 7fe2198b3640 1 -- 172.21.15.47:0/2268383951 >> v1:172.21.15.121:6789/0 conn(0x7fe21413ee90 legacy=0x7fe214131f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:06.231 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:06.254+0000 7fe2198b3640 1 -- 172.21.15.47:0/2268383951 shutdown_connections 2024-03-29T11:20:06.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:06.254+0000 7fe2198b3640 1 -- 172.21.15.47:0/2268383951 >> 172.21.15.47:0/2268383951 conn(0x7fe2140b35e0 msgr2=0x7fe2140b07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:06.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:06.254+0000 7fe2198b3640 1 -- 172.21.15.47:0/2268383951 shutdown_connections 2024-03-29T11:20:06.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:06.254+0000 7fe2198b3640 1 -- 172.21.15.47:0/2268383951 wait complete. 2024-03-29T11:20:06.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:06.254+0000 7fe2198b3640 1 librados: shutdown 2024-03-29T11:20:06.249 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:13.058 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:13.208 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.230+0000 7f7b49f53640 10 monclient: get_monmap_and_config 2024-03-29T11:20:13.208 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.230+0000 7f7b49f53640 10 monclient: build_initial_monmap 2024-03-29T11:20:13.208 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.230+0000 7f7b49f53640 10 monclient: monmap: 2024-03-29T11:20:13.209 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:20:13.209 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:20:13.209 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:20:13.231162+0000 2024-03-29T11:20:13.209 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:20:13.231162+0000 2024-03-29T11:20:13.209 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:13.209 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:20:13.209 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:20:13.209 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:20:13.209 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:20:13.209 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:20:13.209 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.231+0000 7f7b49f53640 1 Processor -- start 2024-03-29T11:20:13.209 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.231+0000 7f7b49f53640 1 -- start start 2024-03-29T11:20:13.209 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.231+0000 7f7b49f53640 10 monclient: init 2024-03-29T11:20:13.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.232+0000 7f7b49f53640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:13.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.232+0000 7f7b49f53640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:20:13.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.232+0000 7f7b49f53640 10 monclient(hunting): picked mon.noname-c con 0x7f7b440a9fe0 addr v1:172.21.15.121:6789/0 2024-03-29T11:20:13.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.232+0000 7f7b49f53640 10 monclient(hunting): picked mon.noname-a con 0x7f7b440af600 addr v1:172.21.15.47:6789/0 2024-03-29T11:20:13.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.232+0000 7f7b49f53640 10 monclient(hunting): picked mon.noname-b con 0x7f7b440ad340 addr v1:172.21.15.47:6790/0 2024-03-29T11:20:13.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.232+0000 7f7b49f53640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b440aaec0 con 0x7f7b440af600 2024-03-29T11:20:13.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.232+0000 7f7b49f53640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b440ab6a0 con 0x7f7b440a9fe0 2024-03-29T11:20:13.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.232+0000 7f7b49f53640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b440b00a0 con 0x7f7b440ad340 2024-03-29T11:20:13.211 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.232+0000 7f7b49f53640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:13.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.232+0000 7f7b49f53640 10 monclient(hunting): authenticate will time out at 1256.420656s 2024-03-29T11:20:13.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.232+0000 7f7b49752640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f7b440ad340 0x7f7b44149c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:42162/0 (socket says 172.21.15.47:42162) 2024-03-29T11:20:13.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.232+0000 7f7b49752640 1 -- 172.21.15.47:0/961449974 learned_addr learned my addr 172.21.15.47:0/961449974 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:20:13.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.232+0000 7f7b43fff640 1 --1- 172.21.15.47:0/961449974 >> v1:172.21.15.47:6789/0 conn(0x7f7b440af600 0x7f7b440acc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:55672/0 (socket says 172.21.15.47:55672) 2024-03-29T11:20:13.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.233+0000 7f7b437fe640 1 -- 172.21.15.47:0/961449974 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4203383880 0 0) 0x7f7b440b00a0 con 0x7f7b440ad340 2024-03-29T11:20:13.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.233+0000 7f7b437fe640 10 monclient(hunting): _negotiate 2024-03-29T11:20:13.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.233+0000 7f7b437fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:13.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.233+0000 7f7b437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:13.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.233+0000 7f7b437fe640 10 monclient(hunting): my global_id is 4520 2024-03-29T11:20:13.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.233+0000 7f7b437fe640 1 -- 172.21.15.47:0/961449974 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b34003440 con 0x7f7b440ad340 2024-03-29T11:20:13.212 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.233+0000 7f7b437fe640 1 -- 172.21.15.47:0/961449974 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2217709455 0 0) 0x7f7b34003440 con 0x7f7b440ad340 2024-03-29T11:20:13.213 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.233+0000 7f7b437fe640 1 -- 172.21.15.47:0/961449974 >> v1:172.21.15.121:6789/0 conn(0x7f7b440a9fe0 legacy=0x7f7b4413b7a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:20:13.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.236+0000 7f7b437fe640 1 -- 172.21.15.47:0/961449974 >> v1:172.21.15.47:6789/0 conn(0x7f7b440af600 legacy=0x7f7b440acc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:13.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.236+0000 7f7b437fe640 10 monclient: _finish_hunting 0 2024-03-29T11:20:13.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.236+0000 7f7b437fe640 1 monclient: found mon.noname-b 2024-03-29T11:20:13.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.236+0000 7f7b437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:13.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.236+0000 7f7b437fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-29T11:20:13.215 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.236+0000 7f7b437fe640 1 -- 172.21.15.47:0/961449974 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b4413beb0 con 0x7f7b440ad340 2024-03-29T11:20:13.216 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.236+0000 7f7b437fe640 10 monclient: _finish_auth 0 2024-03-29T11:20:13.216 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.236+0000 7f7b437fe640 10 monclient: _check_auth_tickets 2024-03-29T11:20:13.216 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.236+0000 7f7b437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:13.216 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.236+0000 7f7b437fe640 1 -- 172.21.15.47:0/961449974 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f7b3c001a10 con 0x7f7b440ad340 2024-03-29T11:20:13.216 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.236+0000 7f7b437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:13.216 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.236+0000 7f7b437fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:13.216 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.236+0000 7f7b437fe640 10 monclient: dump: 2024-03-29T11:20:13.216 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:20:13.216 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:20:13.216 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:13.216 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:13.216 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:13.216 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:20:13.216 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:20:13.217 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:20:13.217 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:20:13.217 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:20:13.217 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.236+0000 7f7b49f53640 5 monclient: authenticate success, global_id 4520 2024-03-29T11:20:13.217 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.236+0000 7f7b49f53640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:13.217 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.237+0000 7f7b437fe640 1 -- 172.21.15.47:0/961449974 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b3c001be0 con 0x7f7b440ad340 2024-03-29T11:20:13.217 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.237+0000 7f7b437fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:13.217 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.237+0000 7f7b437fe640 1 -- 172.21.15.47:0/961449974 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f7b3c002030 con 0x7f7b440ad340 2024-03-29T11:20:13.217 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.237+0000 7f7b437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:13.217 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.237+0000 7f7b437fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:13.217 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.237+0000 7f7b437fe640 10 monclient: dump: 2024-03-29T11:20:13.217 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:20:13.217 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:20:13.217 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:13.217 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:13.218 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:13.218 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:20:13.218 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:20:13.218 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:20:13.218 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:20:13.218 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:20:13.218 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.237+0000 7f7b49f53640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:13.218 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.237+0000 7f7b49f53640 4 set_mon_vals no callback set 2024-03-29T11:20:13.218 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.238+0000 7f7b49f53640 10 monclient: shutdown 2024-03-29T11:20:13.218 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.238+0000 7f7b49f53640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:13.219 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.238+0000 7f7b49f53640 1 -- 172.21.15.47:0/961449974 >> v1:172.21.15.47:6790/0 conn(0x7f7b440ad340 legacy=0x7f7b44149c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:13.219 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.238+0000 7f7b49f53640 1 -- 172.21.15.47:0/961449974 shutdown_connections 2024-03-29T11:20:13.219 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.238+0000 7f7b49f53640 1 -- 172.21.15.47:0/961449974 >> 172.21.15.47:0/961449974 conn(0x7f7b440673e0 msgr2=0x7f7b440b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:13.219 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.238+0000 7f7b49f53640 1 -- 172.21.15.47:0/961449974 shutdown_connections 2024-03-29T11:20:13.219 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.238+0000 7f7b49f53640 1 -- 172.21.15.47:0/961449974 wait complete. 2024-03-29T11:20:13.220 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.239+0000 7f7b49f53640 10 monclient: build_initial_monmap 2024-03-29T11:20:13.220 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.239+0000 7f7b49f53640 10 monclient: monmap: 2024-03-29T11:20:13.220 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:20:13.220 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:13.220 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:20:13.220 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:20:13.220 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:13.220 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:20:13.220 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:20:13.220 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:20:13.221 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:20:13.221 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:20:13.221 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.239+0000 7f7b49f53640 1 librados: starting msgr at 2024-03-29T11:20:13.221 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.239+0000 7f7b49f53640 1 librados: starting objecter 2024-03-29T11:20:13.221 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.239+0000 7f7b49f53640 1 Processor -- start 2024-03-29T11:20:13.221 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.240+0000 7f7b49f53640 1 -- start start 2024-03-29T11:20:13.221 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.240+0000 7f7b49f53640 1 librados: setting wanted keys 2024-03-29T11:20:13.221 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.240+0000 7f7b49f53640 1 librados: calling monclient init 2024-03-29T11:20:13.221 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.240+0000 7f7b49f53640 10 monclient: init 2024-03-29T11:20:13.221 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.241+0000 7f7b49f53640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:13.221 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.241+0000 7f7b49f53640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:20:13.223 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.242+0000 7f7b49f53640 10 monclient(hunting): picked mon.noname-b con 0x7f7b440a9fe0 addr v1:172.21.15.121:6789/0 2024-03-29T11:20:13.223 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.242+0000 7f7b49f53640 10 monclient(hunting): picked mon.noname-a con 0x7f7b440ad340 addr v1:172.21.15.47:6789/0 2024-03-29T11:20:13.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.242+0000 7f7b49f53640 10 monclient(hunting): picked mon.noname-c con 0x7f7b440af600 addr v1:172.21.15.47:6790/0 2024-03-29T11:20:13.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.242+0000 7f7b49f53640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b441d30e0 con 0x7f7b440ad340 2024-03-29T11:20:13.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.242+0000 7f7b49f53640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b441353b0 con 0x7f7b440a9fe0 2024-03-29T11:20:13.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.242+0000 7f7b49f53640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b4413d150 con 0x7f7b440af600 2024-03-29T11:20:13.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.242+0000 7f7b49f53640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:13.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.242+0000 7f7b49f53640 10 monclient(hunting): authenticate will time out at 1256.430411s 2024-03-29T11:20:13.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.242+0000 7f7b43fff640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f7b440ad340 0x7f7b44134590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:56280/0 (socket says 172.21.15.47:56280) 2024-03-29T11:20:13.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.242+0000 7f7b43fff640 1 -- 172.21.15.47:0/1289885440 learned_addr learned my addr 172.21.15.47:0/1289885440 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:20:13.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.242+0000 7f7b417fa640 10 client.?.objecter ms_handle_connect 0x7f7b440ad340 2024-03-29T11:20:13.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.242+0000 7f7b417fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:13.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.242+0000 7f7b417fa640 1 -- 172.21.15.47:0/1289885440 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2565618696 0 0) 0x7f7b441d30e0 con 0x7f7b440ad340 2024-03-29T11:20:13.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.242+0000 7f7b417fa640 10 monclient(hunting): _negotiate 2024-03-29T11:20:13.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.242+0000 7f7b417fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:13.224 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.242+0000 7f7b417fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:13.225 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.242+0000 7f7b417fa640 10 monclient(hunting): my global_id is 4467 2024-03-29T11:20:13.225 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.242+0000 7f7b417fa640 1 -- 172.21.15.47:0/1289885440 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b2800d6c0 con 0x7f7b440ad340 2024-03-29T11:20:13.225 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.243+0000 7f7b417fa640 1 -- 172.21.15.47:0/1289885440 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 33929595 0 0) 0x7f7b2800d6c0 con 0x7f7b440ad340 2024-03-29T11:20:13.226 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.243+0000 7f7b417fa640 1 -- 172.21.15.47:0/1289885440 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b441d30e0 con 0x7f7b440ad340 2024-03-29T11:20:13.226 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.243+0000 7f7b417fa640 1 -- 172.21.15.47:0/1289885440 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f7b30001840 con 0x7f7b440ad340 2024-03-29T11:20:13.226 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.243+0000 7f7b417fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:13.226 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.243+0000 7f7b417fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:13.226 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.243+0000 7f7b417fa640 10 monclient(hunting): dump: 2024-03-29T11:20:13.226 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:20:13.226 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:20:13.227 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:13.227 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:13.227 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:13.227 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:20:13.227 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:20:13.227 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:20:13.227 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:20:13.227 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:20:13.227 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.243+0000 7f7b417fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:13.227 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.243+0000 7f7b417fa640 1 -- 172.21.15.47:0/1289885440 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1014655923 0 0) 0x7f7b441d30e0 con 0x7f7b440ad340 2024-03-29T11:20:13.227 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.243+0000 7f7b417fa640 1 -- 172.21.15.47:0/1289885440 >> v1:172.21.15.47:6790/0 conn(0x7f7b440af600 legacy=0x7f7b44134ca0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:20:13.227 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.243+0000 7f7b417fa640 1 -- 172.21.15.47:0/1289885440 >> v1:172.21.15.121:6789/0 conn(0x7f7b440a9fe0 legacy=0x7f7b441c44f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:20:13.227 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.244+0000 7f7b417fa640 10 monclient: _finish_hunting 0 2024-03-29T11:20:13.227 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.244+0000 7f7b417fa640 1 monclient: found mon.a 2024-03-29T11:20:13.227 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.244+0000 7f7b417fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:13.228 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.244+0000 7f7b417fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:20:13.228 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.244+0000 7f7b417fa640 1 -- 172.21.15.47:0/1289885440 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b4413d430 con 0x7f7b440ad340 2024-03-29T11:20:13.228 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.244+0000 7f7b417fa640 10 monclient: _finish_auth 0 2024-03-29T11:20:13.228 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.244+0000 7f7b417fa640 10 monclient: _check_auth_tickets 2024-03-29T11:20:13.228 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.244+0000 7f7b417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:13.228 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.244+0000 7f7b49f53640 5 monclient: authenticate success, global_id 4467 2024-03-29T11:20:13.228 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.244+0000 7f7b49f53640 10 monclient: _renew_subs 2024-03-29T11:20:13.228 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.244+0000 7f7b49f53640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:20:13.228 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.244+0000 7f7b49f53640 1 -- 172.21.15.47:0/1289885440 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b4413c160 con 0x7f7b440ad340 2024-03-29T11:20:13.228 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.244+0000 7f7b49f53640 10 client.4467.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:13.228 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.244+0000 7f7b49f53640 10 monclient: _renew_subs 2024-03-29T11:20:13.228 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.244+0000 7f7b49f53640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:20:13.228 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.244+0000 7f7b49f53640 1 -- 172.21.15.47:0/1289885440 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b441d2c80 con 0x7f7b440ad340 2024-03-29T11:20:13.228 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.244+0000 7f7b49f53640 1 librados: init done 2024-03-29T11:20:13.228 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.244+0000 7f7b417fa640 1 -- 172.21.15.47:0/1289885440 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b30001aa0 con 0x7f7b440ad340 2024-03-29T11:20:13.229 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.244+0000 7f7b417fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:13.229 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.244+0000 7f7b417fa640 1 -- 172.21.15.47:0/1289885440 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f7b30001d60 con 0x7f7b440ad340 2024-03-29T11:20:13.229 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.244+0000 7f7b417fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:13.229 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.244+0000 7f7b42ffd640 4 set_mon_vals no callback set 2024-03-29T11:20:13.229 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.244+0000 7f7b417fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:13.229 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.244+0000 7f7b417fa640 10 monclient: dump: 2024-03-29T11:20:13.229 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:20:13.229 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:20:13.229 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:13.229 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:13.229 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:13.229 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:20:13.229 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:20:13.229 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:20:13.230 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:20:13.230 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:20:13.230 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.247+0000 7f7b417fa640 1 -- 172.21.15.47:0/1289885440 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7f7b3001aaa0 con 0x7f7b440ad340 2024-03-29T11:20:13.231 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.247+0000 7f7b417fa640 1 -- 172.21.15.47:0/1289885440 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7f7b30084b40 con 0x7f7b440ad340 2024-03-29T11:20:13.231 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.247+0000 7f7b417fa640 10 client.4467.objecter ms_dispatch 0x7f7b44000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:13.231 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.247+0000 7f7b417fa640 3 client.4467.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:13.231 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.247+0000 7f7b417fa640 3 client.4467.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:13.231 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.247+0000 7f7b417fa640 20 client.4467.objecter dump_active .. 0 homeless 2024-03-29T11:20:13.231 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.247+0000 7f7b49f53640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:13.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.247+0000 7f7b49f53640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:13.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.247+0000 7f7b49f53640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:20:13.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.247+0000 7f7b49f53640 1 -- 172.21.15.47:0/1289885440 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b24005080 con 0x7f7b440ad340 2024-03-29T11:20:13.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.248+0000 7f7b48f51640 10 monclient: get_auth_request con 0x7f7b28076420 auth_method 0 2024-03-29T11:20:13.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.248+0000 7f7b417fa640 10 client.4467.objecter ms_handle_connect 0x7f7b28076420 2024-03-29T11:20:13.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.256+0000 7f7b417fa640 1 -- 172.21.15.47:0/1289885440 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7b30050250 con 0x7f7b440ad340 2024-03-29T11:20:13.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.256+0000 7f7b417fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:13.232 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.256+0000 7f7b417fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:13.558 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.582+0000 7f7b49f53640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:13.559 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.582+0000 7f7b49f53640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:13.559 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.582+0000 7f7b49f53640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:20:13.559 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.582+0000 7f7b49f53640 1 -- 172.21.15.47:0/1289885440 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b24005330 con 0x7f7b440ad340 2024-03-29T11:20:13.560 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.583+0000 7f7b417fa640 1 -- 172.21.15.47:0/1289885440 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7f7b30052350 con 0x7f7b440ad340 2024-03-29T11:20:13.560 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.583+0000 7f7b417fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:13.560 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.583+0000 7f7b417fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:13.560 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:20:13.560 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:13.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.587+0000 7f7b49f53640 10 librados: watch_flush enter 2024-03-29T11:20:13.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.587+0000 7f7b49f53640 10 librados: watch_flush exit 2024-03-29T11:20:13.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.587+0000 7f7b49f53640 20 client.4467.objecter shutdown clearing up homeless session... 2024-03-29T11:20:13.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.587+0000 7f7b49f53640 10 client.4467.objecter successfully canceled tick 2024-03-29T11:20:13.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.587+0000 7f7b49f53640 1 -- 172.21.15.47:0/1289885440 >> v1:172.21.15.121:6816/1964117370 conn(0x7f7b28076420 legacy=0x7f7b280788e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:13.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.587+0000 7f7b49f53640 10 monclient: shutdown 2024-03-29T11:20:13.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.587+0000 7f7b49f53640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:13.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.587+0000 7f7b49f53640 1 -- 172.21.15.47:0/1289885440 >> v1:172.21.15.47:6789/0 conn(0x7f7b440ad340 legacy=0x7f7b44134590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:13.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.588+0000 7f7b49f53640 1 -- 172.21.15.47:0/1289885440 shutdown_connections 2024-03-29T11:20:13.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.588+0000 7f7b49f53640 1 -- 172.21.15.47:0/1289885440 >> 172.21.15.47:0/1289885440 conn(0x7f7b440673e0 msgr2=0x7f7b440acbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:13.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.588+0000 7f7b49f53640 1 -- 172.21.15.47:0/1289885440 shutdown_connections 2024-03-29T11:20:13.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.588+0000 7f7b49f53640 1 -- 172.21.15.47:0/1289885440 wait complete. 2024-03-29T11:20:13.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:13.588+0000 7f7b49f53640 1 librados: shutdown 2024-03-29T11:20:13.583 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:20.390 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:20.557 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.580+0000 7fed52025640 10 monclient: get_monmap_and_config 2024-03-29T11:20:20.558 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.580+0000 7fed52025640 10 monclient: build_initial_monmap 2024-03-29T11:20:20.558 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.580+0000 7fed52025640 10 monclient: monmap: 2024-03-29T11:20:20.558 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:20:20.558 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:20:20.558 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:20:20.581881+0000 2024-03-29T11:20:20.558 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:20:20.581881+0000 2024-03-29T11:20:20.558 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:20.558 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:20:20.558 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:20:20.558 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:20:20.558 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:20:20.559 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:20:20.560 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.582+0000 7fed52025640 1 Processor -- start 2024-03-29T11:20:20.560 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.582+0000 7fed52025640 1 -- start start 2024-03-29T11:20:20.560 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.582+0000 7fed52025640 10 monclient: init 2024-03-29T11:20:20.560 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.583+0000 7fed52025640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:20.560 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.583+0000 7fed52025640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:20:20.560 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.583+0000 7fed52025640 10 monclient(hunting): picked mon.noname-b con 0x7fed4c1457f0 addr v1:172.21.15.47:6790/0 2024-03-29T11:20:20.560 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.583+0000 7fed52025640 10 monclient(hunting): picked mon.noname-c con 0x7fed4c13f650 addr v1:172.21.15.121:6789/0 2024-03-29T11:20:20.560 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.583+0000 7fed52025640 10 monclient(hunting): picked mon.noname-a con 0x7fed4c0a9d70 addr v1:172.21.15.47:6789/0 2024-03-29T11:20:20.560 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.583+0000 7fed52025640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed4c0aaef0 con 0x7fed4c0a9d70 2024-03-29T11:20:20.561 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.583+0000 7fed52025640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed4c145bd0 con 0x7fed4c13f650 2024-03-29T11:20:20.561 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.583+0000 7fed52025640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed4c145eb0 con 0x7fed4c1457f0 2024-03-29T11:20:20.561 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.583+0000 7fed52025640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:20.561 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.583+0000 7fed52025640 10 monclient(hunting): authenticate will time out at 1263.771521s 2024-03-29T11:20:20.561 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.583+0000 7fed4affd640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fed4c13f650 0x7fed4c0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.47:37088/0 (socket says 172.21.15.47:37088) 2024-03-29T11:20:20.561 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.583+0000 7fed4affd640 1 -- 172.21.15.47:0/3207349701 learned_addr learned my addr 172.21.15.47:0/3207349701 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:20:20.563 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.584+0000 7fed4a7fc640 1 -- 172.21.15.47:0/3207349701 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3948712911 0 0) 0x7fed4c145bd0 con 0x7fed4c13f650 2024-03-29T11:20:20.563 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.584+0000 7fed4a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:20.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.584+0000 7fed4a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:20.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.584+0000 7fed4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:20.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.584+0000 7fed4a7fc640 10 monclient(hunting): my global_id is 4471 2024-03-29T11:20:20.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.584+0000 7fed4a7fc640 1 -- 172.21.15.47:0/3207349701 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed34003520 con 0x7fed4c13f650 2024-03-29T11:20:20.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.584+0000 7fed4a7fc640 1 -- 172.21.15.47:0/3207349701 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1542814262 0 0) 0x7fed4c145eb0 con 0x7fed4c1457f0 2024-03-29T11:20:20.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.584+0000 7fed4a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:20.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.584+0000 7fed4a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:20.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.584+0000 7fed4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:20.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.584+0000 7fed4a7fc640 10 monclient(hunting): my global_id is 4523 2024-03-29T11:20:20.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.584+0000 7fed4a7fc640 1 -- 172.21.15.47:0/3207349701 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed4c145bd0 con 0x7fed4c1457f0 2024-03-29T11:20:20.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.585+0000 7fed4a7fc640 1 -- 172.21.15.47:0/3207349701 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1268248627 0 0) 0x7fed4c0aaef0 con 0x7fed4c0a9d70 2024-03-29T11:20:20.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.585+0000 7fed4a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:20.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.585+0000 7fed4a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:20.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.585+0000 7fed4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:20.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.585+0000 7fed4a7fc640 10 monclient(hunting): my global_id is 4470 2024-03-29T11:20:20.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.585+0000 7fed4a7fc640 1 -- 172.21.15.47:0/3207349701 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed4c145eb0 con 0x7fed4c0a9d70 2024-03-29T11:20:20.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.585+0000 7fed4a7fc640 1 -- 172.21.15.47:0/3207349701 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3427617814 0 0) 0x7fed34003520 con 0x7fed4c13f650 2024-03-29T11:20:20.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.585+0000 7fed4a7fc640 1 -- 172.21.15.47:0/3207349701 >> v1:172.21.15.47:6790/0 conn(0x7fed4c1457f0 legacy=0x7fed4c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:20.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.585+0000 7fed4a7fc640 1 -- 172.21.15.47:0/3207349701 >> v1:172.21.15.47:6789/0 conn(0x7fed4c0a9d70 legacy=0x7fed4c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:20.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.585+0000 7fed4a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:20:20.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.585+0000 7fed4a7fc640 1 monclient: found mon.noname-c 2024-03-29T11:20:20.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.585+0000 7fed4a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:20.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.585+0000 7fed4a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:20:20.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.585+0000 7fed4a7fc640 1 -- 172.21.15.47:0/3207349701 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed4c146340 con 0x7fed4c13f650 2024-03-29T11:20:20.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.585+0000 7fed4a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:20:20.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.585+0000 7fed4a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:20:20.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.585+0000 7fed4a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:20.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.585+0000 7fed4a7fc640 1 -- 172.21.15.47:0/3207349701 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fed400015c0 con 0x7fed4c13f650 2024-03-29T11:20:20.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.585+0000 7fed4a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:20.566 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.585+0000 7fed4a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:20.566 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.585+0000 7fed4a7fc640 10 monclient: dump: 2024-03-29T11:20:20.566 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:20:20.566 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:20:20.566 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:20.566 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:20.566 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:20.566 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:20:20.566 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:20:20.566 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:20:20.566 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:20:20.566 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:20:20.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.585+0000 7fed52025640 5 monclient: authenticate success, global_id 4471 2024-03-29T11:20:20.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.585+0000 7fed52025640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:20.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.585+0000 7fed4a7fc640 1 -- 172.21.15.47:0/3207349701 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed40001720 con 0x7fed4c13f650 2024-03-29T11:20:20.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.585+0000 7fed4a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:20.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.585+0000 7fed4a7fc640 1 -- 172.21.15.47:0/3207349701 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fed40001b30 con 0x7fed4c13f650 2024-03-29T11:20:20.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.585+0000 7fed4a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:20.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.585+0000 7fed4a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:20.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.585+0000 7fed4a7fc640 10 monclient: dump: 2024-03-29T11:20:20.567 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:20:20.567 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:20:20.567 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:20.567 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:20.567 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:20.567 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:20:20.568 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:20:20.568 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:20:20.568 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:20:20.568 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:20:20.568 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.585+0000 7fed52025640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:20.568 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.585+0000 7fed52025640 4 set_mon_vals no callback set 2024-03-29T11:20:20.568 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.586+0000 7fed52025640 10 monclient: shutdown 2024-03-29T11:20:20.568 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.586+0000 7fed52025640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:20.568 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.586+0000 7fed52025640 1 -- 172.21.15.47:0/3207349701 >> v1:172.21.15.121:6789/0 conn(0x7fed4c13f650 legacy=0x7fed4c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:20.568 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.587+0000 7fed52025640 1 -- 172.21.15.47:0/3207349701 shutdown_connections 2024-03-29T11:20:20.568 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.587+0000 7fed52025640 1 -- 172.21.15.47:0/3207349701 >> 172.21.15.47:0/3207349701 conn(0x7fed4c066a00 msgr2=0x7fed4c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:20.568 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.587+0000 7fed52025640 1 -- 172.21.15.47:0/3207349701 shutdown_connections 2024-03-29T11:20:20.568 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.587+0000 7fed52025640 1 -- 172.21.15.47:0/3207349701 wait complete. 2024-03-29T11:20:20.573 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.588+0000 7fed52025640 10 monclient: build_initial_monmap 2024-03-29T11:20:20.573 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.588+0000 7fed52025640 10 monclient: monmap: 2024-03-29T11:20:20.573 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:20:20.573 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:20.573 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:20:20.574 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:20:20.574 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:20.574 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:20:20.574 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:20:20.574 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:20:20.574 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:20:20.574 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:20:20.574 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.589+0000 7fed52025640 1 librados: starting msgr at 2024-03-29T11:20:20.574 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.589+0000 7fed52025640 1 librados: starting objecter 2024-03-29T11:20:20.574 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.589+0000 7fed52025640 1 Processor -- start 2024-03-29T11:20:20.574 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.589+0000 7fed52025640 1 -- start start 2024-03-29T11:20:20.574 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.589+0000 7fed52025640 1 librados: setting wanted keys 2024-03-29T11:20:20.574 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.589+0000 7fed52025640 1 librados: calling monclient init 2024-03-29T11:20:20.574 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.589+0000 7fed52025640 10 monclient: init 2024-03-29T11:20:20.575 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.590+0000 7fed52025640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:20.575 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.590+0000 7fed52025640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:20:20.575 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.590+0000 7fed52025640 10 monclient(hunting): picked mon.noname-a con 0x7fed4c0a9d70 addr v1:172.21.15.47:6789/0 2024-03-29T11:20:20.575 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.590+0000 7fed52025640 10 monclient(hunting): picked mon.noname-c con 0x7fed4c13f650 addr v1:172.21.15.47:6790/0 2024-03-29T11:20:20.575 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.590+0000 7fed52025640 10 monclient(hunting): picked mon.noname-b con 0x7fed4c1457f0 addr v1:172.21.15.121:6789/0 2024-03-29T11:20:20.575 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.590+0000 7fed52025640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed4c142210 con 0x7fed4c0a9d70 2024-03-29T11:20:20.575 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.590+0000 7fed52025640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed4c13e4b0 con 0x7fed4c1457f0 2024-03-29T11:20:20.575 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.590+0000 7fed52025640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed4c135c10 con 0x7fed4c13f650 2024-03-29T11:20:20.575 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.590+0000 7fed52025640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:20.575 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.590+0000 7fed52025640 10 monclient(hunting): authenticate will time out at 1263.778721s 2024-03-29T11:20:20.575 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.591+0000 7fed4bfff640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fed4c1457f0 0x7fed4c1e1e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.47:37376/0 (socket says 172.21.15.47:37376) 2024-03-29T11:20:20.575 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.591+0000 7fed4bfff640 1 -- 172.21.15.47:0/2486304263 learned_addr learned my addr 172.21.15.47:0/2486304263 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:20:20.575 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.591+0000 7fed33fff640 10 client.?.objecter ms_handle_connect 0x7fed4c1457f0 2024-03-29T11:20:20.575 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.591+0000 7fed33fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:20.575 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.591+0000 7fed33fff640 10 client.?.objecter ms_handle_connect 0x7fed4c0a9d70 2024-03-29T11:20:20.576 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.591+0000 7fed33fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:20.576 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.591+0000 7fed33fff640 10 client.?.objecter ms_handle_connect 0x7fed4c13f650 2024-03-29T11:20:20.576 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.591+0000 7fed33fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:20.576 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.591+0000 7fed33fff640 1 -- 172.21.15.47:0/2486304263 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3997514968 0 0) 0x7fed4c13e4b0 con 0x7fed4c1457f0 2024-03-29T11:20:20.576 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.591+0000 7fed33fff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:20.576 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.591+0000 7fed33fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:20.576 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.591+0000 7fed33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:20.576 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.591+0000 7fed33fff640 10 monclient(hunting): my global_id is 4474 2024-03-29T11:20:20.576 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.591+0000 7fed33fff640 1 -- 172.21.15.47:0/2486304263 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed38004eb0 con 0x7fed4c1457f0 2024-03-29T11:20:20.576 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.592+0000 7fed33fff640 1 -- 172.21.15.47:0/2486304263 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 489576827 0 0) 0x7fed4c135c10 con 0x7fed4c13f650 2024-03-29T11:20:20.576 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.592+0000 7fed33fff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:20.576 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.592+0000 7fed33fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:20.576 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.592+0000 7fed33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:20.576 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.592+0000 7fed33fff640 10 monclient(hunting): my global_id is 4526 2024-03-29T11:20:20.576 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.592+0000 7fed33fff640 1 -- 172.21.15.47:0/2486304263 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed4c13e4b0 con 0x7fed4c13f650 2024-03-29T11:20:20.576 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.592+0000 7fed33fff640 1 -- 172.21.15.47:0/2486304263 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2045868235 0 0) 0x7fed38004eb0 con 0x7fed4c1457f0 2024-03-29T11:20:20.577 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.592+0000 7fed33fff640 1 -- 172.21.15.47:0/2486304263 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed4c135c10 con 0x7fed4c1457f0 2024-03-29T11:20:20.577 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.592+0000 7fed33fff640 1 -- 172.21.15.47:0/2486304263 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1303793213 0 0) 0x7fed4c142210 con 0x7fed4c0a9d70 2024-03-29T11:20:20.577 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.592+0000 7fed33fff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:20.577 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.592+0000 7fed33fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:20.577 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.592+0000 7fed33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:20.577 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.592+0000 7fed33fff640 10 monclient(hunting): my global_id is 4473 2024-03-29T11:20:20.577 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.592+0000 7fed33fff640 1 -- 172.21.15.47:0/2486304263 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed38004eb0 con 0x7fed4c0a9d70 2024-03-29T11:20:20.577 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.592+0000 7fed33fff640 1 -- 172.21.15.47:0/2486304263 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fed44001c90 con 0x7fed4c1457f0 2024-03-29T11:20:20.577 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.592+0000 7fed33fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:20.577 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.592+0000 7fed33fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:20.577 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.592+0000 7fed33fff640 10 monclient(hunting): dump: 2024-03-29T11:20:20.577 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:20:20.577 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:20:20.577 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:20.577 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:20.578 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:20.578 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:20:20.578 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:20:20.578 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:20:20.578 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:20:20.578 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:20:20.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.592+0000 7fed33fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:20.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.592+0000 7fed33fff640 1 -- 172.21.15.47:0/2486304263 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2006767366 0 0) 0x7fed4c135c10 con 0x7fed4c1457f0 2024-03-29T11:20:20.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.592+0000 7fed33fff640 1 -- 172.21.15.47:0/2486304263 >> v1:172.21.15.47:6790/0 conn(0x7fed4c13f650 legacy=0x7fed4c1db0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:20.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.592+0000 7fed33fff640 1 -- 172.21.15.47:0/2486304263 >> v1:172.21.15.47:6789/0 conn(0x7fed4c0a9d70 legacy=0x7fed4c1da990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:20.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.592+0000 7fed33fff640 10 monclient: _finish_hunting 0 2024-03-29T11:20:20.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.592+0000 7fed33fff640 1 monclient: found mon.b 2024-03-29T11:20:20.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.592+0000 7fed33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:20.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.592+0000 7fed33fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:20:20.578 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.592+0000 7fed33fff640 1 -- 172.21.15.47:0/2486304263 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed4c145230 con 0x7fed4c1457f0 2024-03-29T11:20:20.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.592+0000 7fed33fff640 10 monclient: _finish_auth 0 2024-03-29T11:20:20.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.592+0000 7fed33fff640 10 monclient: _check_auth_tickets 2024-03-29T11:20:20.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.592+0000 7fed33fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:20.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.592+0000 7fed52025640 5 monclient: authenticate success, global_id 4474 2024-03-29T11:20:20.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.592+0000 7fed52025640 10 monclient: _renew_subs 2024-03-29T11:20:20.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.592+0000 7fed52025640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:20:20.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.592+0000 7fed52025640 1 -- 172.21.15.47:0/2486304263 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed4c13e0d0 con 0x7fed4c1457f0 2024-03-29T11:20:20.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.592+0000 7fed52025640 10 client.4474.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:20.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.592+0000 7fed52025640 10 monclient: _renew_subs 2024-03-29T11:20:20.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.592+0000 7fed52025640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:20:20.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.592+0000 7fed52025640 1 -- 172.21.15.47:0/2486304263 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fed4c142b60 con 0x7fed4c1457f0 2024-03-29T11:20:20.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.592+0000 7fed52025640 1 librados: init done 2024-03-29T11:20:20.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.593+0000 7fed33fff640 1 -- 172.21.15.47:0/2486304263 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed44001df0 con 0x7fed4c1457f0 2024-03-29T11:20:20.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.593+0000 7fed33fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:20.579 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.593+0000 7fed33fff640 1 -- 172.21.15.47:0/2486304263 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fed44003c10 con 0x7fed4c1457f0 2024-03-29T11:20:20.580 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.593+0000 7fed33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:20.580 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.593+0000 7fed33fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:20.580 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.593+0000 7fed49ffb640 4 set_mon_vals no callback set 2024-03-29T11:20:20.580 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.593+0000 7fed33fff640 10 monclient: dump: 2024-03-29T11:20:20.580 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:20:20.580 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:20:20.580 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:20.580 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:20.580 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:20.580 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:20:20.580 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:20:20.580 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:20:20.580 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:20:20.580 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:20:20.580 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.594+0000 7fed52025640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:20.580 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.595+0000 7fed52025640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:20.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.595+0000 7fed52025640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:20:20.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.595+0000 7fed52025640 1 -- 172.21.15.47:0/2486304263 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed2c005080 con 0x7fed4c1457f0 2024-03-29T11:20:20.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.595+0000 7fed33fff640 1 -- 172.21.15.47:0/2486304263 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7fed44003d70 con 0x7fed4c1457f0 2024-03-29T11:20:20.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.596+0000 7fed33fff640 1 -- 172.21.15.47:0/2486304263 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7fed44084dd0 con 0x7fed4c1457f0 2024-03-29T11:20:20.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.596+0000 7fed33fff640 10 client.4474.objecter ms_dispatch 0x7fed4c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:20.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.596+0000 7fed33fff640 3 client.4474.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:20.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.596+0000 7fed33fff640 3 client.4474.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:20.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.596+0000 7fed33fff640 20 client.4474.objecter dump_active .. 0 homeless 2024-03-29T11:20:20.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.596+0000 7fed4b7fe640 10 monclient: get_auth_request con 0x7fed3806de50 auth_method 0 2024-03-29T11:20:20.581 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.597+0000 7fed33fff640 10 client.4474.objecter ms_handle_connect 0x7fed3806de50 2024-03-29T11:20:20.582 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.602+0000 7fed33fff640 1 -- 172.21.15.47:0/2486304263 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fed440504e0 con 0x7fed4c1457f0 2024-03-29T11:20:20.582 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.602+0000 7fed33fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:20.582 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:20.602+0000 7fed33fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:20.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:21.001+0000 7fed52025640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:20.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:21.001+0000 7fed52025640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:20.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:21.001+0000 7fed52025640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:20:20.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:21.001+0000 7fed52025640 1 -- 172.21.15.47:0/2486304263 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fed2c005330 con 0x7fed4c1457f0 2024-03-29T11:20:20.979 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:21.003+0000 7fed33fff640 1 -- 172.21.15.47:0/2486304263 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7fed440525e0 con 0x7fed4c1457f0 2024-03-29T11:20:20.979 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:21.003+0000 7fed33fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:20.979 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:21.003+0000 7fed33fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:20.980 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:20:20.980 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:20.984 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:21.006+0000 7fed52025640 10 librados: watch_flush enter 2024-03-29T11:20:20.984 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:21.006+0000 7fed52025640 10 librados: watch_flush exit 2024-03-29T11:20:20.984 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:21.006+0000 7fed52025640 20 client.4474.objecter shutdown clearing up homeless session... 2024-03-29T11:20:20.984 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:21.006+0000 7fed52025640 10 client.4474.objecter successfully canceled tick 2024-03-29T11:20:20.984 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:21.007+0000 7fed52025640 1 -- 172.21.15.47:0/2486304263 >> v1:172.21.15.121:6816/1964117370 conn(0x7fed3806de50 legacy=0x7fed38070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:20.984 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:21.007+0000 7fed52025640 10 monclient: shutdown 2024-03-29T11:20:20.985 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:21.007+0000 7fed52025640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:20.985 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:21.007+0000 7fed52025640 1 -- 172.21.15.47:0/2486304263 >> v1:172.21.15.121:6789/0 conn(0x7fed4c1457f0 legacy=0x7fed4c1e1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:20.985 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:21.007+0000 7fed52025640 1 -- 172.21.15.47:0/2486304263 shutdown_connections 2024-03-29T11:20:20.985 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:21.007+0000 7fed52025640 1 -- 172.21.15.47:0/2486304263 >> 172.21.15.47:0/2486304263 conn(0x7fed4c066a00 msgr2=0x7fed4c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:20.985 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:21.007+0000 7fed52025640 1 -- 172.21.15.47:0/2486304263 shutdown_connections 2024-03-29T11:20:20.985 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:21.007+0000 7fed52025640 1 -- 172.21.15.47:0/2486304263 wait complete. 2024-03-29T11:20:20.985 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:21.007+0000 7fed52025640 1 librados: shutdown 2024-03-29T11:20:20.999 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:27.806 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:27.957 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.980+0000 7f35a8625640 10 monclient: get_monmap_and_config 2024-03-29T11:20:27.957 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.980+0000 7f35a8625640 10 monclient: build_initial_monmap 2024-03-29T11:20:27.957 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.980+0000 7f35a8625640 10 monclient: monmap: 2024-03-29T11:20:27.957 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:20:27.957 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:20:27.957 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:20:27.981500+0000 2024-03-29T11:20:27.958 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:20:27.981500+0000 2024-03-29T11:20:27.958 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:27.958 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:20:27.958 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:20:27.958 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:20:27.958 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:20:27.958 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:20:27.958 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.981+0000 7f35a8625640 1 Processor -- start 2024-03-29T11:20:27.963 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.982+0000 7f35a8625640 1 -- start start 2024-03-29T11:20:27.964 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.982+0000 7f35a8625640 10 monclient: init 2024-03-29T11:20:27.964 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.983+0000 7f35a8625640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:27.964 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.983+0000 7f35a8625640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:20:27.964 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.983+0000 7f35a8625640 10 monclient(hunting): picked mon.noname-a con 0x7f35a01422b0 addr v1:172.21.15.47:6789/0 2024-03-29T11:20:27.964 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.983+0000 7f35a8625640 10 monclient(hunting): picked mon.noname-c con 0x7f35a0142690 addr v1:172.21.15.121:6789/0 2024-03-29T11:20:27.964 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.983+0000 7f35a8625640 10 monclient(hunting): picked mon.noname-b con 0x7f35a00b17d0 addr v1:172.21.15.47:6790/0 2024-03-29T11:20:27.964 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.983+0000 7f35a8625640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a0142c80 con 0x7f35a01422b0 2024-03-29T11:20:27.965 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.983+0000 7f35a8625640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a00aae40 con 0x7f35a0142690 2024-03-29T11:20:27.965 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.983+0000 7f35a8625640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a013fbd0 con 0x7f35a00b17d0 2024-03-29T11:20:27.965 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.983+0000 7f35a8625640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:27.965 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.983+0000 7f35a8625640 10 monclient(hunting): authenticate will time out at 1271.171151s 2024-03-29T11:20:27.965 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.983+0000 7f35a63c1640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f35a01422b0 0x7f35a013f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:44482/0 (socket says 172.21.15.47:44482) 2024-03-29T11:20:27.965 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.983+0000 7f35a63c1640 1 -- 172.21.15.47:0/3017614766 learned_addr learned my addr 172.21.15.47:0/3017614766 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:20:27.965 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.983+0000 7f35a53bf640 1 -- 172.21.15.47:0/3017614766 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3492092713 0 0) 0x7f35a0142c80 con 0x7f35a01422b0 2024-03-29T11:20:27.965 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.983+0000 7f35a53bf640 10 monclient(hunting): _negotiate 2024-03-29T11:20:27.965 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.983+0000 7f35a53bf640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:27.965 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.983+0000 7f35a53bf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:27.965 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.983+0000 7f35a53bf640 10 monclient(hunting): my global_id is 4476 2024-03-29T11:20:27.965 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.983+0000 7f35a53bf640 1 -- 172.21.15.47:0/3017614766 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f358c003520 con 0x7f35a01422b0 2024-03-29T11:20:27.965 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.984+0000 7f35a53bf640 1 -- 172.21.15.47:0/3017614766 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1915696156 0 0) 0x7f358c003520 con 0x7f35a01422b0 2024-03-29T11:20:27.966 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.984+0000 7f35a53bf640 1 -- 172.21.15.47:0/3017614766 >> v1:172.21.15.47:6790/0 conn(0x7f35a00b17d0 legacy=0x7f35a00b2ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:20:27.966 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.984+0000 7f35a53bf640 1 -- 172.21.15.47:0/3017614766 >> v1:172.21.15.121:6789/0 conn(0x7f35a0142690 legacy=0x7f35a00b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:20:27.966 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.986+0000 7f35a53bf640 10 monclient: _finish_hunting 0 2024-03-29T11:20:27.966 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.986+0000 7f35a53bf640 1 monclient: found mon.noname-a 2024-03-29T11:20:27.966 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.986+0000 7f35a53bf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:27.966 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.986+0000 7f35a53bf640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-29T11:20:27.966 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.986+0000 7f35a53bf640 1 -- 172.21.15.47:0/3017614766 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35a0151940 con 0x7f35a01422b0 2024-03-29T11:20:27.966 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.986+0000 7f35a53bf640 10 monclient: _finish_auth 0 2024-03-29T11:20:27.966 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.986+0000 7f35a53bf640 10 monclient: _check_auth_tickets 2024-03-29T11:20:27.966 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.986+0000 7f35a53bf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:27.966 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.986+0000 7f35a53bf640 1 -- 172.21.15.47:0/3017614766 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f3598001a10 con 0x7f35a01422b0 2024-03-29T11:20:27.966 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.986+0000 7f35a53bf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:27.966 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.986+0000 7f35a53bf640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:27.966 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.986+0000 7f35a53bf640 10 monclient: dump: 2024-03-29T11:20:27.966 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:20:27.967 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:20:27.967 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:27.967 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:27.967 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:27.967 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:20:27.967 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:20:27.967 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:20:27.967 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:20:27.967 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:20:27.967 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.986+0000 7f35a8625640 5 monclient: authenticate success, global_id 4476 2024-03-29T11:20:27.967 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.986+0000 7f35a8625640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:27.967 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.986+0000 7f35a53bf640 1 -- 172.21.15.47:0/3017614766 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3598001b70 con 0x7f35a01422b0 2024-03-29T11:20:27.967 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.986+0000 7f35a53bf640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:27.967 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.986+0000 7f35a53bf640 1 -- 172.21.15.47:0/3017614766 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f3598001fc0 con 0x7f35a01422b0 2024-03-29T11:20:27.967 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.986+0000 7f35a8625640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:27.968 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.986+0000 7f35a8625640 4 set_mon_vals no callback set 2024-03-29T11:20:27.968 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.986+0000 7f35a53bf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:27.968 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.986+0000 7f35a53bf640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:27.968 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.986+0000 7f35a53bf640 10 monclient: dump: 2024-03-29T11:20:27.968 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:20:27.968 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:20:27.968 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:27.968 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:27.968 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:27.968 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:20:27.968 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:20:27.968 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:20:27.968 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:20:27.968 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:20:27.969 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.987+0000 7f35a8625640 10 monclient: shutdown 2024-03-29T11:20:27.970 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.987+0000 7f35a8625640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:27.970 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.987+0000 7f35a8625640 1 -- 172.21.15.47:0/3017614766 >> v1:172.21.15.47:6789/0 conn(0x7f35a01422b0 legacy=0x7f35a013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:27.970 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.987+0000 7f35a8625640 1 -- 172.21.15.47:0/3017614766 shutdown_connections 2024-03-29T11:20:27.970 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.987+0000 7f35a8625640 1 -- 172.21.15.47:0/3017614766 >> 172.21.15.47:0/3017614766 conn(0x7f35a0066a00 msgr2=0x7f35a0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:27.970 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.991+0000 7f35a8625640 1 -- 172.21.15.47:0/3017614766 shutdown_connections 2024-03-29T11:20:27.970 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.991+0000 7f35a8625640 1 -- 172.21.15.47:0/3017614766 wait complete. 2024-03-29T11:20:27.970 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.992+0000 7f35a8625640 10 monclient: build_initial_monmap 2024-03-29T11:20:27.970 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.992+0000 7f35a8625640 10 monclient: monmap: 2024-03-29T11:20:27.970 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:20:27.970 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:27.970 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:20:27.970 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:20:27.970 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:27.971 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:20:27.971 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:20:27.972 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:20:27.972 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:20:27.972 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:20:27.982 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.992+0000 7f35a8625640 1 librados: starting msgr at 2024-03-29T11:20:27.983 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.992+0000 7f35a8625640 1 librados: starting objecter 2024-03-29T11:20:27.983 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.992+0000 7f35a8625640 1 Processor -- start 2024-03-29T11:20:27.983 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.992+0000 7f35a8625640 1 -- start start 2024-03-29T11:20:27.983 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.992+0000 7f35a8625640 1 librados: setting wanted keys 2024-03-29T11:20:27.983 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.992+0000 7f35a8625640 1 librados: calling monclient init 2024-03-29T11:20:27.983 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.992+0000 7f35a8625640 10 monclient: init 2024-03-29T11:20:27.983 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.993+0000 7f35a8625640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:27.983 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.993+0000 7f35a8625640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:20:27.983 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.993+0000 7f35a8625640 10 monclient(hunting): picked mon.noname-b con 0x7f35a00b17d0 addr v1:172.21.15.121:6789/0 2024-03-29T11:20:27.983 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.993+0000 7f35a8625640 10 monclient(hunting): picked mon.noname-a con 0x7f35a0142690 addr v1:172.21.15.47:6789/0 2024-03-29T11:20:27.983 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.993+0000 7f35a8625640 10 monclient(hunting): picked mon.noname-c con 0x7f35a00b1080 addr v1:172.21.15.47:6790/0 2024-03-29T11:20:27.983 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.993+0000 7f35a8625640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a00abec0 con 0x7f35a0142690 2024-03-29T11:20:27.983 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.993+0000 7f35a8625640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a00acba0 con 0x7f35a00b17d0 2024-03-29T11:20:27.983 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.993+0000 7f35a8625640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a00ace80 con 0x7f35a00b1080 2024-03-29T11:20:27.984 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.993+0000 7f35a8625640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:27.984 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.993+0000 7f35a8625640 10 monclient(hunting): authenticate will time out at 1271.181736s 2024-03-29T11:20:27.984 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.995+0000 7f35a5bc0640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f35a0142690 0x7f35a00b0970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:46318/0 (socket says 172.21.15.47:46318) 2024-03-29T11:20:27.984 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.995+0000 7f35a5bc0640 1 -- 172.21.15.47:0/4177811949 learned_addr learned my addr 172.21.15.47:0/4177811949 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:20:27.985 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.996+0000 7f3596ffd640 10 client.?.objecter ms_handle_connect 0x7f35a0142690 2024-03-29T11:20:27.985 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.996+0000 7f3596ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:27.985 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.996+0000 7f3596ffd640 1 -- 172.21.15.47:0/4177811949 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 972644643 0 0) 0x7f35a00abec0 con 0x7f35a0142690 2024-03-29T11:20:27.985 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.996+0000 7f3596ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:27.985 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.996+0000 7f3596ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:27.985 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.996+0000 7f3596ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:27.985 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.996+0000 7f3596ffd640 10 monclient(hunting): my global_id is 4479 2024-03-29T11:20:27.985 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.996+0000 7f3596ffd640 1 -- 172.21.15.47:0/4177811949 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3584004eb0 con 0x7f35a0142690 2024-03-29T11:20:27.986 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.996+0000 7f3596ffd640 1 -- 172.21.15.47:0/4177811949 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2693753559 0 0) 0x7f3584004eb0 con 0x7f35a0142690 2024-03-29T11:20:27.986 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.996+0000 7f3596ffd640 1 -- 172.21.15.47:0/4177811949 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f35a00abec0 con 0x7f35a0142690 2024-03-29T11:20:27.986 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.996+0000 7f3596ffd640 1 -- 172.21.15.47:0/4177811949 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f359c0017d0 con 0x7f35a0142690 2024-03-29T11:20:27.986 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.996+0000 7f3596ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:27.987 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.996+0000 7f3596ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:27.987 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.996+0000 7f3596ffd640 10 monclient(hunting): dump: 2024-03-29T11:20:27.987 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:20:27.987 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:20:27.987 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:27.987 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:27.987 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:27.988 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:20:27.988 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:20:27.988 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:20:27.988 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:20:27.988 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:20:27.988 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.996+0000 7f3596ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:27.988 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.996+0000 7f3596ffd640 10 client.?.objecter ms_handle_connect 0x7f35a00b1080 2024-03-29T11:20:27.988 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.996+0000 7f3596ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:27.988 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.997+0000 7f3596ffd640 1 -- 172.21.15.47:0/4177811949 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2285006950 0 0) 0x7f35a00ace80 con 0x7f35a00b1080 2024-03-29T11:20:27.988 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.997+0000 7f3596ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:27.988 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.997+0000 7f3596ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:27.988 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.997+0000 7f3596ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:27.988 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.997+0000 7f3596ffd640 10 monclient(hunting): my global_id is 4529 2024-03-29T11:20:27.989 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.997+0000 7f3596ffd640 1 -- 172.21.15.47:0/4177811949 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3584004eb0 con 0x7f35a00b1080 2024-03-29T11:20:27.989 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.997+0000 7f3596ffd640 1 -- 172.21.15.47:0/4177811949 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3347929900 0 0) 0x7f35a00abec0 con 0x7f35a0142690 2024-03-29T11:20:27.989 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.997+0000 7f3596ffd640 1 -- 172.21.15.47:0/4177811949 >> v1:172.21.15.47:6790/0 conn(0x7f35a00b1080 legacy=0x7f35a012fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:27.989 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.997+0000 7f3596ffd640 1 -- 172.21.15.47:0/4177811949 >> v1:172.21.15.121:6789/0 conn(0x7f35a00b17d0 legacy=0x7f35a00ab7b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:20:27.989 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.999+0000 7f3596ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:20:27.989 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.999+0000 7f3596ffd640 1 monclient: found mon.a 2024-03-29T11:20:27.989 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.999+0000 7f3596ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:27.989 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.999+0000 7f3596ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:20:27.989 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.999+0000 7f3596ffd640 1 -- 172.21.15.47:0/4177811949 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35a00ad160 con 0x7f35a0142690 2024-03-29T11:20:27.989 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.999+0000 7f3596ffd640 10 monclient: _finish_auth 0 2024-03-29T11:20:27.989 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.999+0000 7f3596ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:20:27.990 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.999+0000 7f3596ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:27.990 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.999+0000 7f35a8625640 5 monclient: authenticate success, global_id 4479 2024-03-29T11:20:27.990 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.999+0000 7f35a8625640 10 monclient: _renew_subs 2024-03-29T11:20:27.990 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.999+0000 7f35a8625640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:20:27.990 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.999+0000 7f35a8625640 1 -- 172.21.15.47:0/4177811949 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35a0132260 con 0x7f35a0142690 2024-03-29T11:20:27.990 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.999+0000 7f35a8625640 10 client.4479.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:27.990 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.999+0000 7f35a8625640 10 monclient: _renew_subs 2024-03-29T11:20:27.990 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.999+0000 7f35a8625640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:20:27.990 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:27.999+0000 7f35a8625640 1 -- 172.21.15.47:0/4177811949 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f35a0130490 con 0x7f35a0142690 2024-03-29T11:20:27.990 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:28.000+0000 7f35a8625640 1 librados: init done 2024-03-29T11:20:27.990 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:28.002+0000 7f3596ffd640 1 -- 172.21.15.47:0/4177811949 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f359c001930 con 0x7f35a0142690 2024-03-29T11:20:27.991 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:28.002+0000 7f3596ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:27.991 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:28.002+0000 7f3596ffd640 1 -- 172.21.15.47:0/4177811949 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f359c004a00 con 0x7f35a0142690 2024-03-29T11:20:27.991 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:28.002+0000 7f3596ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:27.991 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:28.002+0000 7f3596ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:27.991 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:28.002+0000 7f3596ffd640 10 monclient: dump: 2024-03-29T11:20:27.991 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:20:27.991 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:20:27.991 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:27.991 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:27.991 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:27.992 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:20:27.992 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:20:27.992 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:20:27.992 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:20:27.992 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:20:27.992 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:28.002+0000 7f3596ffd640 1 -- 172.21.15.47:0/4177811949 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7f359c01aa50 con 0x7f35a0142690 2024-03-29T11:20:27.992 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:28.002+0000 7f35a8625640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:27.992 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:28.002+0000 7f35a8625640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:27.992 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:28.002+0000 7f35a8625640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:20:27.993 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:28.002+0000 7f35a8625640 1 -- 172.21.15.47:0/4177811949 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35a00b1490 con 0x7f35a0142690 2024-03-29T11:20:27.993 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:28.002+0000 7f3596ffd640 1 -- 172.21.15.47:0/4177811949 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7f359c084b40 con 0x7f35a0142690 2024-03-29T11:20:27.993 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:28.002+0000 7f3596ffd640 10 client.4479.objecter ms_dispatch 0x7f35a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:27.993 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:28.002+0000 7f3596ffd640 3 client.4479.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:27.993 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:28.002+0000 7f3596ffd640 3 client.4479.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:27.993 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:28.002+0000 7f35a4bbe640 4 set_mon_vals no callback set 2024-03-29T11:20:27.993 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:28.002+0000 7f3596ffd640 20 client.4479.objecter dump_active .. 0 homeless 2024-03-29T11:20:27.993 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:28.003+0000 7f35a63c1640 10 monclient: get_auth_request con 0x7f3584076580 auth_method 0 2024-03-29T11:20:27.993 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:28.003+0000 7f3596ffd640 10 client.4479.objecter ms_handle_connect 0x7f3584076580 2024-03-29T11:20:27.994 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:28.012+0000 7f3596ffd640 1 -- 172.21.15.47:0/4177811949 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f35a00b1490 con 0x7f35a0142690 2024-03-29T11:20:27.994 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:28.012+0000 7f3596ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:27.994 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:28.012+0000 7f3596ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:28.325 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:28.348+0000 7f35a8625640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:28.325 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:28.349+0000 7f35a8625640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:28.326 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:28.349+0000 7f35a8625640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:20:28.326 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:28.349+0000 7f35a8625640 1 -- 172.21.15.47:0/4177811949 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f35a00b1490 con 0x7f35a0142690 2024-03-29T11:20:28.326 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:28.349+0000 7f3596ffd640 1 -- 172.21.15.47:0/4177811949 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7f35a00b1490 con 0x7f35a0142690 2024-03-29T11:20:28.326 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:28.349+0000 7f3596ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:28.326 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:28.349+0000 7f3596ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:28.327 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:20:28.327 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:28.330 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:28.353+0000 7f35a8625640 10 librados: watch_flush enter 2024-03-29T11:20:28.330 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:28.353+0000 7f35a8625640 10 librados: watch_flush exit 2024-03-29T11:20:28.330 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:28.353+0000 7f35a8625640 20 client.4479.objecter shutdown clearing up homeless session... 2024-03-29T11:20:28.330 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:28.353+0000 7f35a8625640 10 client.4479.objecter successfully canceled tick 2024-03-29T11:20:28.331 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:28.355+0000 7f35a8625640 1 -- 172.21.15.47:0/4177811949 >> v1:172.21.15.121:6816/1964117370 conn(0x7f3584076580 legacy=0x7f3584078a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:28.331 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:28.355+0000 7f35a8625640 10 monclient: shutdown 2024-03-29T11:20:28.331 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:28.355+0000 7f35a8625640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:28.331 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:28.355+0000 7f35a8625640 1 -- 172.21.15.47:0/4177811949 >> v1:172.21.15.47:6789/0 conn(0x7f35a0142690 legacy=0x7f35a00b0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:28.331 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:28.355+0000 7f35a8625640 1 -- 172.21.15.47:0/4177811949 shutdown_connections 2024-03-29T11:20:28.332 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:28.355+0000 7f35a8625640 1 -- 172.21.15.47:0/4177811949 >> 172.21.15.47:0/4177811949 conn(0x7f35a0066a00 msgr2=0x7f35a00b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:28.332 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:28.355+0000 7f35a8625640 1 -- 172.21.15.47:0/4177811949 shutdown_connections 2024-03-29T11:20:28.332 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:28.356+0000 7f35a8625640 1 -- 172.21.15.47:0/4177811949 wait complete. 2024-03-29T11:20:28.333 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:28.356+0000 7f35a8625640 1 librados: shutdown 2024-03-29T11:20:28.350 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:35.157 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:35.313 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.335+0000 7f47f4cc5640 10 monclient: get_monmap_and_config 2024-03-29T11:20:35.313 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.335+0000 7f47f4cc5640 10 monclient: build_initial_monmap 2024-03-29T11:20:35.313 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.335+0000 7f47f4cc5640 10 monclient: monmap: 2024-03-29T11:20:35.313 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:20:35.314 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:20:35.314 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:20:35.337861+0000 2024-03-29T11:20:35.314 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:20:35.337861+0000 2024-03-29T11:20:35.314 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:35.314 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:20:35.314 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:20:35.314 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:20:35.314 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:20:35.314 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:20:35.316 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.338+0000 7f47f4cc5640 1 Processor -- start 2024-03-29T11:20:35.316 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.338+0000 7f47f4cc5640 1 -- start start 2024-03-29T11:20:35.316 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.338+0000 7f47f4cc5640 10 monclient: init 2024-03-29T11:20:35.319 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.340+0000 7f47f4cc5640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:35.320 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.340+0000 7f47f4cc5640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:20:35.320 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.340+0000 7f47f4cc5640 10 monclient(hunting): picked mon.noname-c con 0x7f47f01422b0 addr v1:172.21.15.121:6789/0 2024-03-29T11:20:35.320 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.340+0000 7f47f4cc5640 10 monclient(hunting): picked mon.noname-a con 0x7f47f0142690 addr v1:172.21.15.47:6789/0 2024-03-29T11:20:35.320 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.340+0000 7f47f4cc5640 10 monclient(hunting): picked mon.noname-b con 0x7f47f00b17d0 addr v1:172.21.15.47:6790/0 2024-03-29T11:20:35.320 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.342+0000 7f47f4cc5640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47f0142c80 con 0x7f47f0142690 2024-03-29T11:20:35.320 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.342+0000 7f47f4cc5640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47f00aae40 con 0x7f47f01422b0 2024-03-29T11:20:35.320 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.342+0000 7f47f4cc5640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47f013fbd0 con 0x7f47f00b17d0 2024-03-29T11:20:35.320 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.342+0000 7f47f4cc5640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:35.320 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.342+0000 7f47f4cc5640 10 monclient(hunting): authenticate will time out at 1278.531312s 2024-03-29T11:20:35.321 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.343+0000 7f47eed9d640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f47f00b17d0 0x7f47f00b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:58195/0 (socket says 172.21.15.47:58195) 2024-03-29T11:20:35.321 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.343+0000 7f47eed9d640 1 -- 172.21.15.47:0/2979810884 learned_addr learned my addr 172.21.15.47:0/2979810884 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:20:35.322 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.344+0000 7f47ed59a640 1 -- 172.21.15.47:0/2979810884 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1406849246 0 0) 0x7f47f013fbd0 con 0x7f47f00b17d0 2024-03-29T11:20:35.322 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.344+0000 7f47ed59a640 10 monclient(hunting): _negotiate 2024-03-29T11:20:35.323 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.344+0000 7f47ed59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:35.323 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.344+0000 7f47ed59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:35.323 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.344+0000 7f47ed59a640 10 monclient(hunting): my global_id is 4532 2024-03-29T11:20:35.323 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.344+0000 7f47ed59a640 1 -- 172.21.15.47:0/2979810884 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47d8003520 con 0x7f47f00b17d0 2024-03-29T11:20:35.323 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.344+0000 7f47ed59a640 1 -- 172.21.15.47:0/2979810884 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2235096774 0 0) 0x7f47f00aae40 con 0x7f47f01422b0 2024-03-29T11:20:35.323 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.344+0000 7f47ed59a640 10 monclient(hunting): _negotiate 2024-03-29T11:20:35.323 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.344+0000 7f47ed59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:35.323 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.344+0000 7f47ed59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:35.323 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.344+0000 7f47ed59a640 10 monclient(hunting): my global_id is 4477 2024-03-29T11:20:35.323 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.344+0000 7f47ed59a640 1 -- 172.21.15.47:0/2979810884 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47f013fbd0 con 0x7f47f01422b0 2024-03-29T11:20:35.328 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.347+0000 7f47ed59a640 1 -- 172.21.15.47:0/2979810884 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4281771083 0 0) 0x7f47f013fbd0 con 0x7f47f01422b0 2024-03-29T11:20:35.328 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.347+0000 7f47ed59a640 1 -- 172.21.15.47:0/2979810884 >> v1:172.21.15.47:6790/0 conn(0x7f47f00b17d0 legacy=0x7f47f00b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:35.328 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.348+0000 7f47ed59a640 1 -- 172.21.15.47:0/2979810884 >> v1:172.21.15.47:6789/0 conn(0x7f47f0142690 legacy=0x7f47f00b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:20:35.328 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.348+0000 7f47ed59a640 10 monclient: _finish_hunting 0 2024-03-29T11:20:35.328 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.348+0000 7f47ed59a640 1 monclient: found mon.noname-c 2024-03-29T11:20:35.329 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.348+0000 7f47ed59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:35.329 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.348+0000 7f47ed59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:20:35.329 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.348+0000 7f47ed59a640 1 -- 172.21.15.47:0/2979810884 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f0151940 con 0x7f47f01422b0 2024-03-29T11:20:35.329 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.348+0000 7f47ed59a640 10 monclient: _finish_auth 0 2024-03-29T11:20:35.329 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.348+0000 7f47ed59a640 10 monclient: _check_auth_tickets 2024-03-29T11:20:35.329 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.348+0000 7f47ed59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:35.329 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.348+0000 7f47ed59a640 1 -- 172.21.15.47:0/2979810884 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f47e80019f0 con 0x7f47f01422b0 2024-03-29T11:20:35.329 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.348+0000 7f47f4cc5640 5 monclient: authenticate success, global_id 4477 2024-03-29T11:20:35.329 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.348+0000 7f47f4cc5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:35.329 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.348+0000 7f47ed59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:35.329 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.348+0000 7f47ed59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:35.330 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.348+0000 7f47ed59a640 10 monclient: dump: 2024-03-29T11:20:35.330 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:20:35.330 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:20:35.330 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:35.330 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:35.330 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:35.330 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:20:35.330 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:20:35.330 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:20:35.330 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:20:35.330 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:20:35.330 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.349+0000 7f47ed59a640 1 -- 172.21.15.47:0/2979810884 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f47e8001c10 con 0x7f47f01422b0 2024-03-29T11:20:35.330 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.349+0000 7f47f4cc5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:35.330 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.349+0000 7f47ed59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:35.330 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.349+0000 7f47f4cc5640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:35.331 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.349+0000 7f47f4cc5640 4 set_mon_vals no callback set 2024-03-29T11:20:35.331 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.349+0000 7f47ed59a640 1 -- 172.21.15.47:0/2979810884 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f47e8001fa0 con 0x7f47f01422b0 2024-03-29T11:20:35.331 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.349+0000 7f47ed59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:35.331 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.349+0000 7f47ed59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:35.331 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.349+0000 7f47ed59a640 10 monclient: dump: 2024-03-29T11:20:35.331 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:20:35.331 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:20:35.331 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:35.331 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:35.331 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:35.331 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:20:35.331 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:20:35.331 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:20:35.331 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:20:35.332 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:20:35.332 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.350+0000 7f47f4cc5640 10 monclient: shutdown 2024-03-29T11:20:35.332 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.350+0000 7f47f4cc5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:35.332 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.350+0000 7f47f4cc5640 1 -- 172.21.15.47:0/2979810884 >> v1:172.21.15.121:6789/0 conn(0x7f47f01422b0 legacy=0x7f47f013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:35.332 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.350+0000 7f47f4cc5640 1 -- 172.21.15.47:0/2979810884 shutdown_connections 2024-03-29T11:20:35.332 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.350+0000 7f47f4cc5640 1 -- 172.21.15.47:0/2979810884 >> 172.21.15.47:0/2979810884 conn(0x7f47f0066a00 msgr2=0x7f47f0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:35.332 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.353+0000 7f47f4cc5640 1 -- 172.21.15.47:0/2979810884 shutdown_connections 2024-03-29T11:20:35.334 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.354+0000 7f47f4cc5640 1 -- 172.21.15.47:0/2979810884 wait complete. 2024-03-29T11:20:35.334 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.356+0000 7f47f4cc5640 10 monclient: build_initial_monmap 2024-03-29T11:20:35.334 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.356+0000 7f47f4cc5640 10 monclient: monmap: 2024-03-29T11:20:35.334 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:20:35.335 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:35.335 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:20:35.335 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:20:35.335 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:35.335 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:20:35.335 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:20:35.335 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:20:35.335 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:20:35.335 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:20:35.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.356+0000 7f47f4cc5640 1 librados: starting msgr at 2024-03-29T11:20:35.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.356+0000 7f47f4cc5640 1 librados: starting objecter 2024-03-29T11:20:35.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.356+0000 7f47f4cc5640 1 Processor -- start 2024-03-29T11:20:35.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.356+0000 7f47f4cc5640 1 -- start start 2024-03-29T11:20:35.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.356+0000 7f47f4cc5640 1 librados: setting wanted keys 2024-03-29T11:20:35.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.356+0000 7f47f4cc5640 1 librados: calling monclient init 2024-03-29T11:20:35.336 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.356+0000 7f47f4cc5640 10 monclient: init 2024-03-29T11:20:35.336 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.357+0000 7f47f4cc5640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:35.336 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.357+0000 7f47f4cc5640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:20:35.336 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.357+0000 7f47f4cc5640 10 monclient(hunting): picked mon.noname-b con 0x7f47f00b17d0 addr v1:172.21.15.121:6789/0 2024-03-29T11:20:35.336 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.357+0000 7f47f4cc5640 10 monclient(hunting): picked mon.noname-c con 0x7f47f01422b0 addr v1:172.21.15.47:6790/0 2024-03-29T11:20:35.338 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.360+0000 7f47ee59c640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f47f00b17d0 0x7f47f00ab540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.47:44533/0 (socket says 172.21.15.47:44533) 2024-03-29T11:20:35.348 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.364+0000 7f47f4cc5640 10 monclient(hunting): picked mon.noname-a con 0x7f47f0142690 addr v1:172.21.15.47:6789/0 2024-03-29T11:20:35.348 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.364+0000 7f47ee59c640 1 -- 172.21.15.47:0/3379873270 learned_addr learned my addr 172.21.15.47:0/3379873270 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:20:35.348 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.364+0000 7f47f4cc5640 1 -- 172.21.15.47:0/3379873270 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47f01327a0 con 0x7f47f0142690 2024-03-29T11:20:35.348 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.364+0000 7f47deffd640 10 client.?.objecter ms_handle_connect 0x7f47f00b17d0 2024-03-29T11:20:35.349 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.364+0000 7f47deffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:35.349 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.364+0000 7f47deffd640 10 client.?.objecter ms_handle_connect 0x7f47f0142690 2024-03-29T11:20:35.349 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.364+0000 7f47deffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:35.349 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.364+0000 7f47f4cc5640 1 -- 172.21.15.47:0/3379873270 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47f00ac850 con 0x7f47f00b17d0 2024-03-29T11:20:35.349 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.365+0000 7f47f4cc5640 1 -- 172.21.15.47:0/3379873270 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47f00acb30 con 0x7f47f01422b0 2024-03-29T11:20:35.349 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.365+0000 7f47deffd640 1 -- 172.21.15.47:0/3379873270 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3708785849 0 0) 0x7f47f01327a0 con 0x7f47f0142690 2024-03-29T11:20:35.349 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.365+0000 7f47f4cc5640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:35.349 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.365+0000 7f47f4cc5640 10 monclient(hunting): authenticate will time out at 1278.554259s 2024-03-29T11:20:35.349 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.365+0000 7f47deffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:35.349 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.365+0000 7f47deffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:35.349 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.365+0000 7f47deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:35.349 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.365+0000 7f47deffd640 10 monclient(hunting): my global_id is 4482 2024-03-29T11:20:35.349 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.365+0000 7f47deffd640 1 -- 172.21.15.47:0/3379873270 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47d0004eb0 con 0x7f47f0142690 2024-03-29T11:20:35.349 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.365+0000 7f47deffd640 1 -- 172.21.15.47:0/3379873270 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1224948957 0 0) 0x7f47f00ac850 con 0x7f47f00b17d0 2024-03-29T11:20:35.350 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.365+0000 7f47deffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:35.350 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.365+0000 7f47deffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:35.350 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.365+0000 7f47deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:35.350 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.365+0000 7f47deffd640 10 monclient(hunting): my global_id is 4480 2024-03-29T11:20:35.350 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.365+0000 7f47deffd640 1 -- 172.21.15.47:0/3379873270 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47f01327a0 con 0x7f47f00b17d0 2024-03-29T11:20:35.350 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.366+0000 7f47deffd640 1 -- 172.21.15.47:0/3379873270 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4145666557 0 0) 0x7f47d0004eb0 con 0x7f47f0142690 2024-03-29T11:20:35.350 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.366+0000 7f47deffd640 1 -- 172.21.15.47:0/3379873270 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47f00ac850 con 0x7f47f0142690 2024-03-29T11:20:35.350 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.366+0000 7f47deffd640 10 client.?.objecter ms_handle_connect 0x7f47f01422b0 2024-03-29T11:20:35.350 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.366+0000 7f47deffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:35.350 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.366+0000 7f47deffd640 1 -- 172.21.15.47:0/3379873270 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f47e4001a70 con 0x7f47f0142690 2024-03-29T11:20:35.350 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.366+0000 7f47deffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:35.351 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.366+0000 7f47deffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:35.351 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.366+0000 7f47deffd640 10 monclient(hunting): dump: 2024-03-29T11:20:35.351 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:20:35.351 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:20:35.351 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:35.351 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:35.351 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:35.351 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:20:35.351 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:20:35.351 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:20:35.351 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:20:35.351 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:20:35.351 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.366+0000 7f47deffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:35.351 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.366+0000 7f47deffd640 1 -- 172.21.15.47:0/3379873270 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3815302123 0 0) 0x7f47f00acb30 con 0x7f47f01422b0 2024-03-29T11:20:35.352 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.366+0000 7f47deffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:35.352 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.366+0000 7f47deffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:35.352 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.366+0000 7f47deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:35.352 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.366+0000 7f47deffd640 10 monclient(hunting): my global_id is 4535 2024-03-29T11:20:35.352 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.366+0000 7f47deffd640 1 -- 172.21.15.47:0/3379873270 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47d0004eb0 con 0x7f47f01422b0 2024-03-29T11:20:35.352 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.367+0000 7f47deffd640 1 -- 172.21.15.47:0/3379873270 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3038871627 0 0) 0x7f47f01327a0 con 0x7f47f00b17d0 2024-03-29T11:20:35.352 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.367+0000 7f47deffd640 1 -- 172.21.15.47:0/3379873270 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47f00acb30 con 0x7f47f00b17d0 2024-03-29T11:20:35.352 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.367+0000 7f47deffd640 1 -- 172.21.15.47:0/3379873270 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f47e80019b0 con 0x7f47f00b17d0 2024-03-29T11:20:35.352 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.367+0000 7f47deffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:35.352 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.367+0000 7f47deffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:35.352 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.367+0000 7f47deffd640 10 monclient(hunting): dump: 2024-03-29T11:20:35.352 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:20:35.352 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:20:35.352 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:35.353 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:35.353 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:35.353 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:20:35.353 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:20:35.353 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:20:35.353 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:20:35.353 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:20:35.353 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.367+0000 7f47deffd640 1 -- 172.21.15.47:0/3379873270 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1593227157 0 0) 0x7f47f00ac850 con 0x7f47f0142690 2024-03-29T11:20:35.353 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.367+0000 7f47deffd640 1 -- 172.21.15.47:0/3379873270 >> v1:172.21.15.47:6790/0 conn(0x7f47f01422b0 legacy=0x7f47f012fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:35.353 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.367+0000 7f47deffd640 1 -- 172.21.15.47:0/3379873270 >> v1:172.21.15.121:6789/0 conn(0x7f47f00b17d0 legacy=0x7f47f00ab540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:35.353 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.367+0000 7f47deffd640 10 monclient: _finish_hunting 0 2024-03-29T11:20:35.353 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.367+0000 7f47deffd640 1 monclient: found mon.a 2024-03-29T11:20:35.353 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.367+0000 7f47deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:35.353 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.367+0000 7f47deffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:20:35.353 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.367+0000 7f47deffd640 1 -- 172.21.15.47:0/3379873270 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f00ace10 con 0x7f47f0142690 2024-03-29T11:20:35.354 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.367+0000 7f47deffd640 10 monclient: _finish_auth 0 2024-03-29T11:20:35.354 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.367+0000 7f47deffd640 10 monclient: _check_auth_tickets 2024-03-29T11:20:35.354 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.367+0000 7f47deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:35.354 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.368+0000 7f47f4cc5640 5 monclient: authenticate success, global_id 4482 2024-03-29T11:20:35.354 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.368+0000 7f47f4cc5640 10 monclient: _renew_subs 2024-03-29T11:20:35.354 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.368+0000 7f47deffd640 1 -- 172.21.15.47:0/3379873270 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f47e4001c50 con 0x7f47f0142690 2024-03-29T11:20:35.354 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.368+0000 7f47f4cc5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:20:35.354 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.368+0000 7f47f4cc5640 1 -- 172.21.15.47:0/3379873270 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47f01329f0 con 0x7f47f0142690 2024-03-29T11:20:35.354 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.368+0000 7f47deffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:35.354 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.368+0000 7f47deffd640 1 -- 172.21.15.47:0/3379873270 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f47e4004c30 con 0x7f47f0142690 2024-03-29T11:20:35.354 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.368+0000 7f47deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:35.355 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.368+0000 7f47deffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:35.355 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.368+0000 7f47deffd640 10 monclient: dump: 2024-03-29T11:20:35.355 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:20:35.355 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:20:35.355 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:35.355 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:35.355 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:35.355 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:20:35.355 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:20:35.355 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:20:35.355 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:20:35.355 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:20:35.355 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.368+0000 7f47ecd99640 4 set_mon_vals no callback set 2024-03-29T11:20:35.355 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.368+0000 7f47f4cc5640 10 client.4482.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:35.356 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.368+0000 7f47f4cc5640 10 monclient: _renew_subs 2024-03-29T11:20:35.356 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.368+0000 7f47f4cc5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:20:35.356 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.368+0000 7f47f4cc5640 1 -- 172.21.15.47:0/3379873270 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f47f01e2e70 con 0x7f47f0142690 2024-03-29T11:20:35.356 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.368+0000 7f47f4cc5640 1 librados: init done 2024-03-29T11:20:35.356 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.370+0000 7f47f4cc5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:35.356 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.370+0000 7f47f4cc5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:35.356 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.370+0000 7f47f4cc5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:20:35.356 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.370+0000 7f47f4cc5640 1 -- 172.21.15.47:0/3379873270 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47f0132560 con 0x7f47f0142690 2024-03-29T11:20:35.357 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.374+0000 7f47deffd640 1 -- 172.21.15.47:0/3379873270 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7f47e401ac00 con 0x7f47f0142690 2024-03-29T11:20:35.357 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.374+0000 7f47deffd640 1 -- 172.21.15.47:0/3379873270 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7f47e40849c0 con 0x7f47f0142690 2024-03-29T11:20:35.357 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.375+0000 7f47deffd640 10 client.4482.objecter ms_dispatch 0x7f47f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:35.357 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.375+0000 7f47deffd640 3 client.4482.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:35.357 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.375+0000 7f47deffd640 3 client.4482.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:35.357 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.375+0000 7f47deffd640 20 client.4482.objecter dump_active .. 0 homeless 2024-03-29T11:20:35.357 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.375+0000 7f47ee59c640 10 monclient: get_auth_request con 0x7f47d006e040 auth_method 0 2024-03-29T11:20:35.358 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.376+0000 7f47deffd640 10 client.4482.objecter ms_handle_connect 0x7f47d006e040 2024-03-29T11:20:35.361 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.384+0000 7f47deffd640 1 -- 172.21.15.47:0/3379873270 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f47e40502a0 con 0x7f47f0142690 2024-03-29T11:20:35.361 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.384+0000 7f47deffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:35.362 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.384+0000 7f47deffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:35.694 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.717+0000 7f47f4cc5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:35.694 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.717+0000 7f47f4cc5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:35.694 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.717+0000 7f47f4cc5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:20:35.695 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.717+0000 7f47f4cc5640 1 -- 172.21.15.47:0/3379873270 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f47f0075f10 con 0x7f47f0142690 2024-03-29T11:20:35.696 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.719+0000 7f47deffd640 1 -- 172.21.15.47:0/3379873270 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7f47e40521d0 con 0x7f47f0142690 2024-03-29T11:20:35.696 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.719+0000 7f47deffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:35.696 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.719+0000 7f47deffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:35.698 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:20:35.698 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:35.704 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.726+0000 7f47f4cc5640 10 librados: watch_flush enter 2024-03-29T11:20:35.704 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.726+0000 7f47f4cc5640 10 librados: watch_flush exit 2024-03-29T11:20:35.704 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.726+0000 7f47f4cc5640 20 client.4482.objecter shutdown clearing up homeless session... 2024-03-29T11:20:35.704 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.726+0000 7f47f4cc5640 10 client.4482.objecter successfully canceled tick 2024-03-29T11:20:35.704 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.726+0000 7f47f4cc5640 1 -- 172.21.15.47:0/3379873270 >> v1:172.21.15.121:6816/1964117370 conn(0x7f47d006e040 legacy=0x7f47d0070500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:35.704 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.726+0000 7f47f4cc5640 10 monclient: shutdown 2024-03-29T11:20:35.704 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.726+0000 7f47f4cc5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:35.704 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.726+0000 7f47f4cc5640 1 -- 172.21.15.47:0/3379873270 >> v1:172.21.15.47:6789/0 conn(0x7f47f0142690 legacy=0x7f47f0130540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:35.705 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.726+0000 7f47f4cc5640 1 -- 172.21.15.47:0/3379873270 shutdown_connections 2024-03-29T11:20:35.705 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.726+0000 7f47f4cc5640 1 -- 172.21.15.47:0/3379873270 >> 172.21.15.47:0/3379873270 conn(0x7f47f0066a00 msgr2=0x7f47f00b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:35.705 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.727+0000 7f47f4cc5640 1 -- 172.21.15.47:0/3379873270 shutdown_connections 2024-03-29T11:20:35.705 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.727+0000 7f47f4cc5640 1 -- 172.21.15.47:0/3379873270 wait complete. 2024-03-29T11:20:35.705 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:35.727+0000 7f47f4cc5640 1 librados: shutdown 2024-03-29T11:20:35.722 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:42.529 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:42.683 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.705+0000 7f8c4b59e640 10 monclient: get_monmap_and_config 2024-03-29T11:20:42.683 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.705+0000 7f8c4b59e640 10 monclient: build_initial_monmap 2024-03-29T11:20:42.683 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.705+0000 7f8c4b59e640 10 monclient: monmap: 2024-03-29T11:20:42.683 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:20:42.684 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:20:42.684 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:20:42.706621+0000 2024-03-29T11:20:42.684 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:20:42.706621+0000 2024-03-29T11:20:42.684 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:42.684 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:20:42.684 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:20:42.684 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:20:42.684 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:20:42.685 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:20:42.687 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.710+0000 7f8c4b59e640 1 Processor -- start 2024-03-29T11:20:42.687 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.710+0000 7f8c4b59e640 1 -- start start 2024-03-29T11:20:42.687 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.710+0000 7f8c4b59e640 10 monclient: init 2024-03-29T11:20:42.689 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.711+0000 7f8c4b59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:42.689 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.711+0000 7f8c4b59e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:20:42.690 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.711+0000 7f8c4b59e640 10 monclient(hunting): picked mon.noname-c con 0x7f8c44006890 addr v1:172.21.15.121:6789/0 2024-03-29T11:20:42.690 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.711+0000 7f8c4b59e640 10 monclient(hunting): picked mon.noname-a con 0x7f8c44006c70 addr v1:172.21.15.47:6789/0 2024-03-29T11:20:42.690 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.711+0000 7f8c4b59e640 10 monclient(hunting): picked mon.noname-b con 0x7f8c4402b4c0 addr v1:172.21.15.47:6790/0 2024-03-29T11:20:42.690 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.711+0000 7f8c4b59e640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c4402c830 con 0x7f8c44006c70 2024-03-29T11:20:42.690 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.711+0000 7f8c4b59e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c4402afa0 con 0x7f8c44006890 2024-03-29T11:20:42.691 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.711+0000 7f8c4b59e640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c4402aba0 con 0x7f8c4402b4c0 2024-03-29T11:20:42.691 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.711+0000 7f8c4b59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:42.691 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.712+0000 7f8c4b59e640 10 monclient(hunting): authenticate will time out at 1285.899969s 2024-03-29T11:20:42.692 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.715+0000 7f8c4a59c640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f8c44006890 0x7f8c440c0a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.47:56583/0 (socket says 172.21.15.47:56583) 2024-03-29T11:20:42.692 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.715+0000 7f8c4a59c640 1 -- 172.21.15.47:0/2544675024 learned_addr learned my addr 172.21.15.47:0/2544675024 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:20:42.695 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.719+0000 7f8c4959a640 1 -- 172.21.15.47:0/2544675024 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3233516506 0 0) 0x7f8c4402aba0 con 0x7f8c4402b4c0 2024-03-29T11:20:42.695 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.719+0000 7f8c4959a640 10 monclient(hunting): _negotiate 2024-03-29T11:20:42.696 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.719+0000 7f8c4959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:42.696 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.719+0000 7f8c4959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:42.696 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.719+0000 7f8c4959a640 10 monclient(hunting): my global_id is 4538 2024-03-29T11:20:42.696 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.719+0000 7f8c4959a640 1 -- 172.21.15.47:0/2544675024 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c34003520 con 0x7f8c4402b4c0 2024-03-29T11:20:42.702 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.719+0000 7f8c4959a640 1 -- 172.21.15.47:0/2544675024 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2224521780 0 0) 0x7f8c34003520 con 0x7f8c4402b4c0 2024-03-29T11:20:42.703 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.719+0000 7f8c4959a640 1 -- 172.21.15.47:0/2544675024 >> v1:172.21.15.121:6789/0 conn(0x7f8c44006890 legacy=0x7f8c440c0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:42.703 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.719+0000 7f8c4959a640 1 -- 172.21.15.47:0/2544675024 >> v1:172.21.15.47:6789/0 conn(0x7f8c44006c70 legacy=0x7f8c440c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:20:42.703 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.719+0000 7f8c4959a640 10 monclient: _finish_hunting 0 2024-03-29T11:20:42.703 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.719+0000 7f8c4959a640 1 monclient: found mon.noname-b 2024-03-29T11:20:42.703 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.719+0000 7f8c4959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:42.703 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.719+0000 7f8c4959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-29T11:20:42.703 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.719+0000 7f8c4959a640 1 -- 172.21.15.47:0/2544675024 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c4402ad30 con 0x7f8c4402b4c0 2024-03-29T11:20:42.703 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.719+0000 7f8c4959a640 10 monclient: _finish_auth 0 2024-03-29T11:20:42.703 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.719+0000 7f8c4959a640 10 monclient: _check_auth_tickets 2024-03-29T11:20:42.703 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.719+0000 7f8c4959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:42.703 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.719+0000 7f8c4959a640 1 -- 172.21.15.47:0/2544675024 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f8c4c04f730 con 0x7f8c4402b4c0 2024-03-29T11:20:42.703 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.719+0000 7f8c4959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:42.703 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.719+0000 7f8c4959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:42.703 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.719+0000 7f8c4959a640 10 monclient: dump: 2024-03-29T11:20:42.704 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:20:42.704 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:20:42.704 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:42.704 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:42.704 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:42.704 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:20:42.704 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:20:42.704 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:20:42.704 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:20:42.704 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:20:42.704 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.719+0000 7f8c4b59e640 5 monclient: authenticate success, global_id 4538 2024-03-29T11:20:42.704 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.719+0000 7f8c4b59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:42.704 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.720+0000 7f8c4959a640 1 -- 172.21.15.47:0/2544675024 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c4c0008d0 con 0x7f8c4402b4c0 2024-03-29T11:20:42.704 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.720+0000 7f8c4959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:42.705 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.720+0000 7f8c4959a640 1 -- 172.21.15.47:0/2544675024 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f8c4c04f730 con 0x7f8c4402b4c0 2024-03-29T11:20:42.705 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.720+0000 7f8c4959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:42.705 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.720+0000 7f8c4959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:42.705 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.720+0000 7f8c4959a640 10 monclient: dump: 2024-03-29T11:20:42.705 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:20:42.705 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:20:42.705 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:42.705 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:42.705 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:42.705 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:20:42.705 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:20:42.705 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:20:42.705 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:20:42.705 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:20:42.705 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.720+0000 7f8c4b59e640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:42.705 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.720+0000 7f8c4b59e640 4 set_mon_vals no callback set 2024-03-29T11:20:42.706 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.721+0000 7f8c4b59e640 10 monclient: shutdown 2024-03-29T11:20:42.706 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.721+0000 7f8c4b59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:42.706 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.721+0000 7f8c4b59e640 1 -- 172.21.15.47:0/2544675024 >> v1:172.21.15.47:6790/0 conn(0x7f8c4402b4c0 legacy=0x7f8c440cf0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:42.706 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.723+0000 7f8c4b59e640 1 -- 172.21.15.47:0/2544675024 shutdown_connections 2024-03-29T11:20:42.706 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.723+0000 7f8c4b59e640 1 -- 172.21.15.47:0/2544675024 >> 172.21.15.47:0/2544675024 conn(0x7f8c440058f0 msgr2=0x7f8c44005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:42.706 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.723+0000 7f8c4b59e640 1 -- 172.21.15.47:0/2544675024 shutdown_connections 2024-03-29T11:20:42.706 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.723+0000 7f8c4b59e640 1 -- 172.21.15.47:0/2544675024 wait complete. 2024-03-29T11:20:42.706 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.725+0000 7f8c4b59e640 10 monclient: build_initial_monmap 2024-03-29T11:20:42.706 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.725+0000 7f8c4b59e640 10 monclient: monmap: 2024-03-29T11:20:42.706 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:20:42.706 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:42.706 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:20:42.706 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:20:42.706 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:42.707 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:20:42.707 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:20:42.707 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:20:42.707 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:20:42.707 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:20:42.707 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.726+0000 7f8c4b59e640 1 librados: starting msgr at 2024-03-29T11:20:42.707 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.726+0000 7f8c4b59e640 1 librados: starting objecter 2024-03-29T11:20:42.707 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.726+0000 7f8c4b59e640 1 Processor -- start 2024-03-29T11:20:42.707 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.726+0000 7f8c4b59e640 1 -- start start 2024-03-29T11:20:42.708 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.726+0000 7f8c4b59e640 1 librados: setting wanted keys 2024-03-29T11:20:42.708 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.726+0000 7f8c4b59e640 1 librados: calling monclient init 2024-03-29T11:20:42.708 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.726+0000 7f8c4b59e640 10 monclient: init 2024-03-29T11:20:42.708 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.727+0000 7f8c4b59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:42.708 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.727+0000 7f8c4b59e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:20:42.708 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.727+0000 7f8c4b59e640 10 monclient(hunting): picked mon.noname-a con 0x7f8c44006890 addr v1:172.21.15.47:6789/0 2024-03-29T11:20:42.708 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.727+0000 7f8c4b59e640 10 monclient(hunting): picked mon.noname-b con 0x7f8c4402b4c0 addr v1:172.21.15.121:6789/0 2024-03-29T11:20:42.708 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.727+0000 7f8c4b59e640 10 monclient(hunting): picked mon.noname-c con 0x7f8c440c2490 addr v1:172.21.15.47:6790/0 2024-03-29T11:20:42.708 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.727+0000 7f8c4b59e640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c440c48f0 con 0x7f8c44006890 2024-03-29T11:20:42.708 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.727+0000 7f8c4b59e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c44031d80 con 0x7f8c4402b4c0 2024-03-29T11:20:42.708 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.727+0000 7f8c4b59e640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c44160cb0 con 0x7f8c440c2490 2024-03-29T11:20:42.708 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.727+0000 7f8c4b59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:42.708 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.727+0000 7f8c4b59e640 10 monclient(hunting): authenticate will time out at 1285.915541s 2024-03-29T11:20:42.711 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.735+0000 7f8c4ad9d640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f8c440c2490 0x7f8c440c2ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:40055/0 (socket says 172.21.15.47:40055) 2024-03-29T11:20:42.712 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.735+0000 7f8c4ad9d640 1 -- 172.21.15.47:0/2414540311 learned_addr learned my addr 172.21.15.47:0/2414540311 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:20:42.712 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.735+0000 7f8c3affd640 10 client.?.objecter ms_handle_connect 0x7f8c440c2490 2024-03-29T11:20:42.712 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.735+0000 7f8c3affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:42.715 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.736+0000 7f8c3affd640 10 client.?.objecter ms_handle_connect 0x7f8c44006890 2024-03-29T11:20:42.715 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.736+0000 7f8c3affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:42.715 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.736+0000 7f8c3affd640 1 -- 172.21.15.47:0/2414540311 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2151281050 0 0) 0x7f8c44160cb0 con 0x7f8c440c2490 2024-03-29T11:20:42.715 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.736+0000 7f8c3affd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:42.715 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.736+0000 7f8c3affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:42.715 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.736+0000 7f8c3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:42.715 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.736+0000 7f8c3affd640 10 monclient(hunting): my global_id is 4541 2024-03-29T11:20:42.715 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.736+0000 7f8c3affd640 1 -- 172.21.15.47:0/2414540311 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c2c004eb0 con 0x7f8c440c2490 2024-03-29T11:20:42.716 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.736+0000 7f8c3affd640 1 -- 172.21.15.47:0/2414540311 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2892574470 0 0) 0x7f8c440c48f0 con 0x7f8c44006890 2024-03-29T11:20:42.716 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.736+0000 7f8c3affd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:42.716 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.736+0000 7f8c3affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:42.716 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.736+0000 7f8c3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:42.716 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.736+0000 7f8c3affd640 10 monclient(hunting): my global_id is 4485 2024-03-29T11:20:42.716 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.736+0000 7f8c3affd640 1 -- 172.21.15.47:0/2414540311 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c44160cb0 con 0x7f8c44006890 2024-03-29T11:20:42.716 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.738+0000 7f8c3affd640 1 -- 172.21.15.47:0/2414540311 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3852962610 0 0) 0x7f8c2c004eb0 con 0x7f8c440c2490 2024-03-29T11:20:42.716 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.738+0000 7f8c3affd640 1 -- 172.21.15.47:0/2414540311 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c440c48f0 con 0x7f8c440c2490 2024-03-29T11:20:42.716 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.738+0000 7f8c3affd640 1 -- 172.21.15.47:0/2414540311 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f8c4c04f730 con 0x7f8c440c2490 2024-03-29T11:20:42.716 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.738+0000 7f8c3affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:42.716 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.738+0000 7f8c3affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:42.716 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.738+0000 7f8c3affd640 10 monclient(hunting): dump: 2024-03-29T11:20:42.716 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:20:42.716 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:20:42.717 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:42.717 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:42.717 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:42.717 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:20:42.717 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:20:42.717 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:20:42.717 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:20:42.717 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:20:42.721 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.738+0000 7f8c3affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:42.721 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.740+0000 7f8c3affd640 10 client.?.objecter ms_handle_connect 0x7f8c4402b4c0 2024-03-29T11:20:42.721 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.740+0000 7f8c3affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:42.721 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.740+0000 7f8c3affd640 1 -- 172.21.15.47:0/2414540311 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3203566888 0 0) 0x7f8c44160cb0 con 0x7f8c44006890 2024-03-29T11:20:42.721 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.740+0000 7f8c3affd640 1 -- 172.21.15.47:0/2414540311 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c2c006900 con 0x7f8c44006890 2024-03-29T11:20:42.721 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.740+0000 7f8c3affd640 1 -- 172.21.15.47:0/2414540311 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 766861159 0 0) 0x7f8c440c48f0 con 0x7f8c440c2490 2024-03-29T11:20:42.721 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.740+0000 7f8c3affd640 1 -- 172.21.15.47:0/2414540311 >> v1:172.21.15.121:6789/0 conn(0x7f8c4402b4c0 legacy=0x7f8c441611d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:42.721 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.740+0000 7f8c3affd640 1 -- 172.21.15.47:0/2414540311 >> v1:172.21.15.47:6789/0 conn(0x7f8c44006890 legacy=0x7f8c440bc1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:42.721 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.740+0000 7f8c3affd640 10 monclient: _finish_hunting 0 2024-03-29T11:20:42.722 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.740+0000 7f8c3affd640 1 monclient: found mon.c 2024-03-29T11:20:42.722 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.740+0000 7f8c3affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:42.722 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.740+0000 7f8c3affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:20:42.722 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.740+0000 7f8c3affd640 1 -- 172.21.15.47:0/2414540311 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c440be130 con 0x7f8c440c2490 2024-03-29T11:20:42.722 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.740+0000 7f8c3affd640 10 monclient: _finish_auth 0 2024-03-29T11:20:42.722 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.740+0000 7f8c3affd640 10 monclient: _check_auth_tickets 2024-03-29T11:20:42.722 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.740+0000 7f8c3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:42.722 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.741+0000 7f8c4b59e640 5 monclient: authenticate success, global_id 4541 2024-03-29T11:20:42.722 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.741+0000 7f8c4b59e640 10 monclient: _renew_subs 2024-03-29T11:20:42.722 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.741+0000 7f8c4b59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:20:42.722 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.742+0000 7f8c4b59e640 1 -- 172.21.15.47:0/2414540311 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c441618e0 con 0x7f8c440c2490 2024-03-29T11:20:42.722 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.742+0000 7f8c3affd640 1 -- 172.21.15.47:0/2414540311 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c4c067bf0 con 0x7f8c440c2490 2024-03-29T11:20:42.722 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.742+0000 7f8c3affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:42.723 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.742+0000 7f8c3affd640 1 -- 172.21.15.47:0/2414540311 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f8c4c04f730 con 0x7f8c440c2490 2024-03-29T11:20:42.723 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.742+0000 7f8c3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:42.723 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.742+0000 7f8c3affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:42.723 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.742+0000 7f8c3affd640 10 monclient: dump: 2024-03-29T11:20:42.723 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:20:42.723 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:20:42.723 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:42.723 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:42.723 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:42.723 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:20:42.723 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:20:42.723 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:20:42.723 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:20:42.723 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:20:42.723 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.742+0000 7f8c4b59e640 10 client.4541.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:42.724 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.742+0000 7f8c4b59e640 10 monclient: _renew_subs 2024-03-29T11:20:42.724 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.742+0000 7f8c4b59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:20:42.724 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.742+0000 7f8c4b59e640 1 -- 172.21.15.47:0/2414540311 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c440c37b0 con 0x7f8c440c2490 2024-03-29T11:20:42.724 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.742+0000 7f8c4b59e640 1 librados: init done 2024-03-29T11:20:42.724 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.743+0000 7f8c48d99640 4 set_mon_vals no callback set 2024-03-29T11:20:42.724 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.744+0000 7f8c4b59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:42.724 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.744+0000 7f8c4b59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:42.724 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.744+0000 7f8c4b59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:20:42.724 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.744+0000 7f8c4b59e640 1 -- 172.21.15.47:0/2414540311 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c30005080 con 0x7f8c440c2490 2024-03-29T11:20:42.725 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.747+0000 7f8c3affd640 1 -- 172.21.15.47:0/2414540311 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7f8c4c067d60 con 0x7f8c440c2490 2024-03-29T11:20:42.725 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.748+0000 7f8c3affd640 1 -- 172.21.15.47:0/2414540311 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7f8c4c0cf0c0 con 0x7f8c440c2490 2024-03-29T11:20:42.725 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.748+0000 7f8c3affd640 10 client.4541.objecter ms_dispatch 0x7f8c4c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:42.726 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.748+0000 7f8c3affd640 3 client.4541.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:42.726 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.748+0000 7f8c3affd640 3 client.4541.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:42.726 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.748+0000 7f8c3affd640 20 client.4541.objecter dump_active .. 0 homeless 2024-03-29T11:20:42.726 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.749+0000 7f8c4a59c640 10 monclient: get_auth_request con 0x7f8c2c06df40 auth_method 0 2024-03-29T11:20:42.726 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.749+0000 7f8c3affd640 10 client.4541.objecter ms_handle_connect 0x7f8c2c06df40 2024-03-29T11:20:42.733 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.756+0000 7f8c3affd640 1 -- 172.21.15.47:0/2414540311 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8c4c0cb360 con 0x7f8c440c2490 2024-03-29T11:20:42.735 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.756+0000 7f8c3affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:42.735 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:42.756+0000 7f8c3affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:43.058 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:43.082+0000 7f8c4b59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:43.059 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:43.082+0000 7f8c4b59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:43.059 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:43.082+0000 7f8c4b59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:20:43.059 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:43.082+0000 7f8c4b59e640 1 -- 172.21.15.47:0/2414540311 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8c30005330 con 0x7f8c440c2490 2024-03-29T11:20:43.060 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:43.084+0000 7f8c3affd640 1 -- 172.21.15.47:0/2414540311 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7f8c4c0cb360 con 0x7f8c440c2490 2024-03-29T11:20:43.060 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:43.084+0000 7f8c3affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:43.060 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:43.084+0000 7f8c3affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:43.061 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:20:43.061 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:43.064 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:43.087+0000 7f8c4b59e640 10 librados: watch_flush enter 2024-03-29T11:20:43.064 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:43.087+0000 7f8c4b59e640 10 librados: watch_flush exit 2024-03-29T11:20:43.064 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:43.087+0000 7f8c4b59e640 20 client.4541.objecter shutdown clearing up homeless session... 2024-03-29T11:20:43.064 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:43.087+0000 7f8c4b59e640 10 client.4541.objecter successfully canceled tick 2024-03-29T11:20:43.064 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:43.087+0000 7f8c4b59e640 1 -- 172.21.15.47:0/2414540311 >> v1:172.21.15.121:6816/1964117370 conn(0x7f8c2c06df40 legacy=0x7f8c2c070400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:43.064 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:43.087+0000 7f8c4b59e640 10 monclient: shutdown 2024-03-29T11:20:43.064 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:43.087+0000 7f8c4b59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:43.064 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:43.087+0000 7f8c4b59e640 1 -- 172.21.15.47:0/2414540311 >> v1:172.21.15.47:6790/0 conn(0x7f8c440c2490 legacy=0x7f8c440c2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:43.065 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:43.089+0000 7f8c4b59e640 1 -- 172.21.15.47:0/2414540311 shutdown_connections 2024-03-29T11:20:43.065 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:43.089+0000 7f8c4b59e640 1 -- 172.21.15.47:0/2414540311 >> 172.21.15.47:0/2414540311 conn(0x7f8c440058f0 msgr2=0x7f8c44038130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:43.067 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:43.090+0000 7f8c4b59e640 1 -- 172.21.15.47:0/2414540311 shutdown_connections 2024-03-29T11:20:43.067 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:43.090+0000 7f8c4b59e640 1 -- 172.21.15.47:0/2414540311 wait complete. 2024-03-29T11:20:43.067 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:43.091+0000 7f8c4b59e640 1 librados: shutdown 2024-03-29T11:20:43.084 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:49.892 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:50.043 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.066+0000 7f5bcf49a640 10 monclient: get_monmap_and_config 2024-03-29T11:20:50.043 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.066+0000 7f5bcf49a640 10 monclient: build_initial_monmap 2024-03-29T11:20:50.043 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.066+0000 7f5bcf49a640 10 monclient: monmap: 2024-03-29T11:20:50.043 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:20:50.043 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:20:50.043 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:20:50.067316+0000 2024-03-29T11:20:50.043 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:20:50.067316+0000 2024-03-29T11:20:50.043 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:50.043 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:20:50.043 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:20:50.044 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:20:50.044 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:20:50.044 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:20:50.045 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.068+0000 7f5bcf49a640 1 Processor -- start 2024-03-29T11:20:50.046 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.069+0000 7f5bcf49a640 1 -- start start 2024-03-29T11:20:50.046 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.069+0000 7f5bcf49a640 10 monclient: init 2024-03-29T11:20:50.048 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.070+0000 7f5bcf49a640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:50.048 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.070+0000 7f5bcf49a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:20:50.048 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.070+0000 7f5bcf49a640 10 monclient(hunting): picked mon.noname-a con 0x7f5bc80b0770 addr v1:172.21.15.47:6789/0 2024-03-29T11:20:50.048 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.070+0000 7f5bcf49a640 10 monclient(hunting): picked mon.noname-c con 0x7f5bc814a700 addr v1:172.21.15.121:6789/0 2024-03-29T11:20:50.048 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.070+0000 7f5bcf49a640 10 monclient(hunting): picked mon.noname-b con 0x7f5bc80ac5a0 addr v1:172.21.15.47:6790/0 2024-03-29T11:20:50.048 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.070+0000 7f5bcf49a640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bc80aba70 con 0x7f5bc80b0770 2024-03-29T11:20:50.048 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.070+0000 7f5bcf49a640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bc80aecb0 con 0x7f5bc814a700 2024-03-29T11:20:50.048 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.070+0000 7f5bcf49a640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bc80af150 con 0x7f5bc80ac5a0 2024-03-29T11:20:50.048 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.070+0000 7f5bcf49a640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:50.048 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.070+0000 7f5bcf49a640 10 monclient(hunting): authenticate will time out at 1293.258779s 2024-03-29T11:20:50.050 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.073+0000 7f5bcda37640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f5bc80ac5a0 0x7f5bc8151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:34766/0 (socket says 172.21.15.47:34766) 2024-03-29T11:20:50.050 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.073+0000 7f5bcda37640 1 -- 172.21.15.47:0/925227571 learned_addr learned my addr 172.21.15.47:0/925227571 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:20:50.050 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.074+0000 7f5bbffff640 1 -- 172.21.15.47:0/925227571 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2688456679 0 0) 0x7f5bc80af150 con 0x7f5bc80ac5a0 2024-03-29T11:20:50.050 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.074+0000 7f5bbffff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:50.050 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.074+0000 7f5bbffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:50.051 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.074+0000 7f5bbffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:50.051 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.074+0000 7f5bbffff640 10 monclient(hunting): my global_id is 4544 2024-03-29T11:20:50.051 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.074+0000 7f5bbffff640 1 -- 172.21.15.47:0/925227571 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bb4003520 con 0x7f5bc80ac5a0 2024-03-29T11:20:50.051 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.074+0000 7f5bbffff640 1 -- 172.21.15.47:0/925227571 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3892674227 0 0) 0x7f5bb4003520 con 0x7f5bc80ac5a0 2024-03-29T11:20:50.051 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.074+0000 7f5bbffff640 1 -- 172.21.15.47:0/925227571 >> v1:172.21.15.121:6789/0 conn(0x7f5bc814a700 legacy=0x7f5bc80abe90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:20:50.055 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.077+0000 7f5bbffff640 1 -- 172.21.15.47:0/925227571 >> v1:172.21.15.47:6789/0 conn(0x7f5bc80b0770 legacy=0x7f5bc8149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:50.055 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.077+0000 7f5bbffff640 10 monclient: _finish_hunting 0 2024-03-29T11:20:50.055 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.077+0000 7f5bbffff640 1 monclient: found mon.noname-b 2024-03-29T11:20:50.055 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.077+0000 7f5bbffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:50.055 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.077+0000 7f5bbffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-29T11:20:50.055 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.077+0000 7f5bbffff640 1 -- 172.21.15.47:0/925227571 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bc80ac9d0 con 0x7f5bc80ac5a0 2024-03-29T11:20:50.055 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.077+0000 7f5bbffff640 10 monclient: _finish_auth 0 2024-03-29T11:20:50.055 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.077+0000 7f5bbffff640 10 monclient: _check_auth_tickets 2024-03-29T11:20:50.055 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.077+0000 7f5bbffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:50.055 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.078+0000 7f5bcf49a640 5 monclient: authenticate success, global_id 4544 2024-03-29T11:20:50.056 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.078+0000 7f5bcf49a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:50.056 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.078+0000 7f5bbffff640 1 -- 172.21.15.47:0/925227571 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f5bc0001a10 con 0x7f5bc80ac5a0 2024-03-29T11:20:50.056 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.078+0000 7f5bbffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:50.056 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.078+0000 7f5bbffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:50.056 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.078+0000 7f5bbffff640 10 monclient: dump: 2024-03-29T11:20:50.056 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:20:50.056 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:20:50.056 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:50.056 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:50.056 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:50.056 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:20:50.056 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:20:50.056 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:20:50.056 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:20:50.056 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:20:50.057 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.078+0000 7f5bcf49a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:50.057 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.079+0000 7f5bbffff640 1 -- 172.21.15.47:0/925227571 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5bc0001ce0 con 0x7f5bc80ac5a0 2024-03-29T11:20:50.057 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.079+0000 7f5bbffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:50.057 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.079+0000 7f5bbffff640 1 -- 172.21.15.47:0/925227571 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f5bc0002070 con 0x7f5bc80ac5a0 2024-03-29T11:20:50.057 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.079+0000 7f5bbffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:50.057 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.079+0000 7f5bbffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:50.057 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.079+0000 7f5bbffff640 10 monclient: dump: 2024-03-29T11:20:50.058 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:20:50.058 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:20:50.058 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:50.058 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:50.058 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:50.058 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:20:50.058 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:20:50.059 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:20:50.059 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:20:50.059 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:20:50.059 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.080+0000 7f5bcf49a640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:50.059 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.080+0000 7f5bcf49a640 4 set_mon_vals no callback set 2024-03-29T11:20:50.059 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.081+0000 7f5bcf49a640 10 monclient: shutdown 2024-03-29T11:20:50.060 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.081+0000 7f5bcf49a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:50.060 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.081+0000 7f5bcf49a640 1 -- 172.21.15.47:0/925227571 >> v1:172.21.15.47:6790/0 conn(0x7f5bc80ac5a0 legacy=0x7f5bc8151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:50.060 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.082+0000 7f5bcf49a640 1 -- 172.21.15.47:0/925227571 shutdown_connections 2024-03-29T11:20:50.060 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.082+0000 7f5bcf49a640 1 -- 172.21.15.47:0/925227571 >> 172.21.15.47:0/925227571 conn(0x7f5bc806b660 msgr2=0x7f5bc806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:50.060 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.082+0000 7f5bcf49a640 1 -- 172.21.15.47:0/925227571 shutdown_connections 2024-03-29T11:20:50.060 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.082+0000 7f5bcf49a640 1 -- 172.21.15.47:0/925227571 wait complete. 2024-03-29T11:20:50.062 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.083+0000 7f5bcf49a640 10 monclient: build_initial_monmap 2024-03-29T11:20:50.062 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.083+0000 7f5bcf49a640 10 monclient: monmap: 2024-03-29T11:20:50.062 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:20:50.062 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:50.062 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:20:50.062 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:20:50.062 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:50.062 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:20:50.062 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:20:50.062 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:20:50.063 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:20:50.063 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:20:50.063 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.084+0000 7f5bcf49a640 1 librados: starting msgr at 2024-03-29T11:20:50.063 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.084+0000 7f5bcf49a640 1 librados: starting objecter 2024-03-29T11:20:50.063 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.084+0000 7f5bcf49a640 1 Processor -- start 2024-03-29T11:20:50.063 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.084+0000 7f5bcf49a640 1 -- start start 2024-03-29T11:20:50.063 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.084+0000 7f5bcf49a640 1 librados: setting wanted keys 2024-03-29T11:20:50.063 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.084+0000 7f5bcf49a640 1 librados: calling monclient init 2024-03-29T11:20:50.063 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.084+0000 7f5bcf49a640 10 monclient: init 2024-03-29T11:20:50.063 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.085+0000 7f5bcf49a640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:50.063 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.085+0000 7f5bcf49a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:20:50.063 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.085+0000 7f5bcf49a640 10 monclient(hunting): picked mon.noname-b con 0x7f5bc80ac5a0 addr v1:172.21.15.121:6789/0 2024-03-29T11:20:50.063 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.085+0000 7f5bcf49a640 10 monclient(hunting): picked mon.noname-c con 0x7f5bc80b0770 addr v1:172.21.15.47:6790/0 2024-03-29T11:20:50.063 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.085+0000 7f5bcf49a640 10 monclient(hunting): picked mon.noname-a con 0x7f5bc814a700 addr v1:172.21.15.47:6789/0 2024-03-29T11:20:50.064 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.085+0000 7f5bcf49a640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bc81275b0 con 0x7f5bc814a700 2024-03-29T11:20:50.064 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.085+0000 7f5bcf49a640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bc8127890 con 0x7f5bc80ac5a0 2024-03-29T11:20:50.064 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.085+0000 7f5bcf49a640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bc8126660 con 0x7f5bc80b0770 2024-03-29T11:20:50.064 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.085+0000 7f5bcf49a640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:50.064 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.085+0000 7f5bcf49a640 10 monclient(hunting): authenticate will time out at 1293.273108s 2024-03-29T11:20:50.064 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.085+0000 7f5bcca35640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f5bc80b0770 0x7f5bc8125400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:38808/0 (socket says 172.21.15.47:38808) 2024-03-29T11:20:50.064 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.085+0000 7f5bcca35640 1 -- 172.21.15.47:0/2862874372 learned_addr learned my addr 172.21.15.47:0/2862874372 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:20:50.064 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.085+0000 7f5bbdffb640 10 client.?.objecter ms_handle_connect 0x7f5bc80b0770 2024-03-29T11:20:50.064 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.085+0000 7f5bbdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:50.064 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.085+0000 7f5bbdffb640 1 -- 172.21.15.47:0/2862874372 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2847450591 0 0) 0x7f5bc8126660 con 0x7f5bc80b0770 2024-03-29T11:20:50.064 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.085+0000 7f5bbdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:20:50.064 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.085+0000 7f5bbdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:50.064 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.085+0000 7f5bbdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:50.065 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.085+0000 7f5bbdffb640 10 monclient(hunting): my global_id is 4547 2024-03-29T11:20:50.065 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.085+0000 7f5bbdffb640 1 -- 172.21.15.47:0/2862874372 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bac004eb0 con 0x7f5bc80b0770 2024-03-29T11:20:50.065 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.086+0000 7f5bbdffb640 1 -- 172.21.15.47:0/2862874372 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2257887442 0 0) 0x7f5bac004eb0 con 0x7f5bc80b0770 2024-03-29T11:20:50.065 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.086+0000 7f5bbdffb640 1 -- 172.21.15.47:0/2862874372 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5bc8126660 con 0x7f5bc80b0770 2024-03-29T11:20:50.065 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.086+0000 7f5bbdffb640 1 -- 172.21.15.47:0/2862874372 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f5bb80017d0 con 0x7f5bc80b0770 2024-03-29T11:20:50.065 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.086+0000 7f5bbdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:50.066 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.086+0000 7f5bbdffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:50.066 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.086+0000 7f5bbdffb640 10 monclient(hunting): dump: 2024-03-29T11:20:50.066 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:20:50.066 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:20:50.066 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:50.066 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:50.066 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:50.066 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:20:50.066 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:20:50.066 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:20:50.066 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:20:50.066 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:20:50.066 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.086+0000 7f5bbdffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:50.066 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.086+0000 7f5bbdffb640 1 -- 172.21.15.47:0/2862874372 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1874482900 0 0) 0x7f5bc8126660 con 0x7f5bc80b0770 2024-03-29T11:20:50.067 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.086+0000 7f5bbdffb640 1 -- 172.21.15.47:0/2862874372 >> v1:172.21.15.121:6789/0 conn(0x7f5bc80ac5a0 legacy=0x7f5bc8147590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:20:50.067 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.088+0000 7f5bbdffb640 1 -- 172.21.15.47:0/2862874372 >> v1:172.21.15.47:6789/0 conn(0x7f5bc814a700 legacy=0x7f5bc8125b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:20:50.068 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.090+0000 7f5bbdffb640 10 monclient: _finish_hunting 0 2024-03-29T11:20:50.068 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.091+0000 7f5bbdffb640 1 monclient: found mon.c 2024-03-29T11:20:50.068 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.091+0000 7f5bbdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:50.068 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.091+0000 7f5bbdffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:20:50.068 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.091+0000 7f5bbdffb640 1 -- 172.21.15.47:0/2862874372 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bc8126220 con 0x7f5bc80b0770 2024-03-29T11:20:50.068 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.091+0000 7f5bbdffb640 10 monclient: _finish_auth 0 2024-03-29T11:20:50.068 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.091+0000 7f5bbdffb640 10 monclient: _check_auth_tickets 2024-03-29T11:20:50.068 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.091+0000 7f5bbdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:50.068 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.091+0000 7f5bbdffb640 1 -- 172.21.15.47:0/2862874372 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5bb8001930 con 0x7f5bc80b0770 2024-03-29T11:20:50.071 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.091+0000 7f5bcf49a640 5 monclient: authenticate success, global_id 4547 2024-03-29T11:20:50.071 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.091+0000 7f5bcf49a640 10 monclient: _renew_subs 2024-03-29T11:20:50.071 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.091+0000 7f5bcf49a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:20:50.072 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.091+0000 7f5bcf49a640 1 -- 172.21.15.47:0/2862874372 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bc8147ca0 con 0x7f5bc80b0770 2024-03-29T11:20:50.072 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.091+0000 7f5bcf49a640 10 client.4547.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:50.072 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.091+0000 7f5bcf49a640 10 monclient: _renew_subs 2024-03-29T11:20:50.072 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.091+0000 7f5bcf49a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:20:50.072 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.091+0000 7f5bcf49a640 1 -- 172.21.15.47:0/2862874372 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bc81dea30 con 0x7f5bc80b0770 2024-03-29T11:20:50.072 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.091+0000 7f5bcf49a640 1 librados: init done 2024-03-29T11:20:50.072 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.091+0000 7f5bbdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:50.072 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.091+0000 7f5bbf7fe640 4 set_mon_vals no callback set 2024-03-29T11:20:50.072 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.092+0000 7f5bbdffb640 1 -- 172.21.15.47:0/2862874372 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f5bb8004a40 con 0x7f5bc80b0770 2024-03-29T11:20:50.072 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.092+0000 7f5bbdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:50.072 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.092+0000 7f5bbdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:50.072 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.092+0000 7f5bbdffb640 10 monclient: dump: 2024-03-29T11:20:50.072 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:20:50.072 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:20:50.073 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:50.073 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:50.073 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:50.073 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:20:50.073 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:20:50.073 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:20:50.073 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:20:50.073 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:20:50.073 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.093+0000 7f5bbdffb640 1 -- 172.21.15.47:0/2862874372 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7f5bb801aad0 con 0x7f5bc80b0770 2024-03-29T11:20:50.073 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.093+0000 7f5bcf49a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:50.073 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.093+0000 7f5bcf49a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:50.073 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.093+0000 7f5bcf49a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:20:50.073 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.094+0000 7f5bbdffb640 1 -- 172.21.15.47:0/2862874372 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7f5bb8084aa0 con 0x7f5bc80b0770 2024-03-29T11:20:50.073 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.094+0000 7f5bbdffb640 10 client.4547.objecter ms_dispatch 0x7f5bc8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:50.073 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.094+0000 7f5bbdffb640 3 client.4547.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:50.074 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.094+0000 7f5bbdffb640 3 client.4547.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:50.074 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.094+0000 7f5bbdffb640 20 client.4547.objecter dump_active .. 0 homeless 2024-03-29T11:20:50.074 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.093+0000 7f5bcf49a640 1 -- 172.21.15.47:0/2862874372 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ba8005080 con 0x7f5bc80b0770 2024-03-29T11:20:50.074 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.094+0000 7f5bcd236640 10 monclient: get_auth_request con 0x7f5bac06dc80 auth_method 0 2024-03-29T11:20:50.075 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.098+0000 7f5bbdffb640 10 client.4547.objecter ms_handle_connect 0x7f5bac06dc80 2024-03-29T11:20:50.079 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.102+0000 7f5bbdffb640 1 -- 172.21.15.47:0/2862874372 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5bb80501b0 con 0x7f5bc80b0770 2024-03-29T11:20:50.079 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.103+0000 7f5bbdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:50.079 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.103+0000 7f5bbdffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:50.415 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.438+0000 7f5bcf49a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:50.415 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.438+0000 7f5bcf49a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:50.415 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.438+0000 7f5bcf49a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:20:50.415 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.438+0000 7f5bcf49a640 1 -- 172.21.15.47:0/2862874372 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5ba8005330 con 0x7f5bc80b0770 2024-03-29T11:20:50.417 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.441+0000 7f5bbdffb640 1 -- 172.21.15.47:0/2862874372 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7f5bb80522b0 con 0x7f5bc80b0770 2024-03-29T11:20:50.417 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.441+0000 7f5bbdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:50.417 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.441+0000 7f5bbdffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:50.419 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:20:50.419 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:50.421 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.444+0000 7f5bcf49a640 10 librados: watch_flush enter 2024-03-29T11:20:50.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.446+0000 7f5bcf49a640 10 librados: watch_flush exit 2024-03-29T11:20:50.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.446+0000 7f5bcf49a640 20 client.4547.objecter shutdown clearing up homeless session... 2024-03-29T11:20:50.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.446+0000 7f5bcf49a640 10 client.4547.objecter successfully canceled tick 2024-03-29T11:20:50.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.446+0000 7f5bcf49a640 1 -- 172.21.15.47:0/2862874372 >> v1:172.21.15.121:6816/1964117370 conn(0x7f5bac06dc80 legacy=0x7f5bac070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:50.423 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.446+0000 7f5bcf49a640 10 monclient: shutdown 2024-03-29T11:20:50.423 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.446+0000 7f5bcf49a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:50.423 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.446+0000 7f5bcf49a640 1 -- 172.21.15.47:0/2862874372 >> v1:172.21.15.47:6790/0 conn(0x7f5bc80b0770 legacy=0x7f5bc8125400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:50.423 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.446+0000 7f5bcf49a640 1 -- 172.21.15.47:0/2862874372 shutdown_connections 2024-03-29T11:20:50.423 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.446+0000 7f5bcf49a640 1 -- 172.21.15.47:0/2862874372 >> 172.21.15.47:0/2862874372 conn(0x7f5bc806b660 msgr2=0x7f5bc80ae2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:50.424 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.447+0000 7f5bcf49a640 1 -- 172.21.15.47:0/2862874372 shutdown_connections 2024-03-29T11:20:50.424 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.447+0000 7f5bcf49a640 1 -- 172.21.15.47:0/2862874372 wait complete. 2024-03-29T11:20:50.424 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:50.447+0000 7f5bcf49a640 1 librados: shutdown 2024-03-29T11:20:50.441 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:57.248 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:57.400 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.423+0000 7fc2addfd640 10 monclient: get_monmap_and_config 2024-03-29T11:20:57.400 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.423+0000 7fc2addfd640 10 monclient: build_initial_monmap 2024-03-29T11:20:57.400 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.423+0000 7fc2addfd640 10 monclient: monmap: 2024-03-29T11:20:57.400 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:20:57.400 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:20:57.400 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:20:57.424712+0000 2024-03-29T11:20:57.400 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:20:57.424712+0000 2024-03-29T11:20:57.401 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:57.401 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:20:57.401 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:20:57.401 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:20:57.401 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:20:57.401 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:20:57.401 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.425+0000 7fc2addfd640 1 Processor -- start 2024-03-29T11:20:57.402 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.425+0000 7fc2addfd640 1 -- start start 2024-03-29T11:20:57.402 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.425+0000 7fc2addfd640 10 monclient: init 2024-03-29T11:20:57.403 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.426+0000 7fc2addfd640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:57.403 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.426+0000 7fc2addfd640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:20:57.404 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.426+0000 7fc2addfd640 10 monclient(hunting): picked mon.noname-b con 0x7fc2a81457f0 addr v1:172.21.15.47:6790/0 2024-03-29T11:20:57.404 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.426+0000 7fc2addfd640 10 monclient(hunting): picked mon.noname-a con 0x7fc2a813f650 addr v1:172.21.15.47:6789/0 2024-03-29T11:20:57.404 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.426+0000 7fc2addfd640 10 monclient(hunting): picked mon.noname-c con 0x7fc2a80a9d70 addr v1:172.21.15.121:6789/0 2024-03-29T11:20:57.404 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.426+0000 7fc2addfd640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2a80aaef0 con 0x7fc2a813f650 2024-03-29T11:20:57.404 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.426+0000 7fc2addfd640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2a8145bd0 con 0x7fc2a80a9d70 2024-03-29T11:20:57.404 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.426+0000 7fc2addfd640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2a8145eb0 con 0x7fc2a81457f0 2024-03-29T11:20:57.404 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.426+0000 7fc2addfd640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:57.404 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.426+0000 7fc2addfd640 10 monclient(hunting): authenticate will time out at 1300.614392s 2024-03-29T11:20:57.404 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.427+0000 7fc2a77fe640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fc2a81457f0 0x7fc2a813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:34270/0 (socket says 172.21.15.47:34270) 2024-03-29T11:20:57.404 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.427+0000 7fc2a77fe640 1 -- 172.21.15.47:0/2039509819 learned_addr learned my addr 172.21.15.47:0/2039509819 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:20:57.405 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.428+0000 7fc2a67fc640 1 -- 172.21.15.47:0/2039509819 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1512790430 0 0) 0x7fc2a8145eb0 con 0x7fc2a81457f0 2024-03-29T11:20:57.405 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.428+0000 7fc2a67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:57.405 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.428+0000 7fc2a67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:57.405 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.428+0000 7fc2a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:57.405 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.428+0000 7fc2a67fc640 10 monclient(hunting): my global_id is 4550 2024-03-29T11:20:57.406 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.428+0000 7fc2a67fc640 1 -- 172.21.15.47:0/2039509819 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc290003520 con 0x7fc2a81457f0 2024-03-29T11:20:57.406 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.428+0000 7fc2a67fc640 1 -- 172.21.15.47:0/2039509819 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2292426687 0 0) 0x7fc290003520 con 0x7fc2a81457f0 2024-03-29T11:20:57.406 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.428+0000 7fc2a67fc640 1 -- 172.21.15.47:0/2039509819 >> v1:172.21.15.121:6789/0 conn(0x7fc2a80a9d70 legacy=0x7fc2a814d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:20:57.408 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.430+0000 7fc2a67fc640 1 -- 172.21.15.47:0/2039509819 >> v1:172.21.15.47:6789/0 conn(0x7fc2a813f650 legacy=0x7fc2a80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:57.408 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.430+0000 7fc2a67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:20:57.408 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.430+0000 7fc2a67fc640 1 monclient: found mon.noname-b 2024-03-29T11:20:57.408 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.430+0000 7fc2a67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:57.408 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.430+0000 7fc2a67fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-29T11:20:57.408 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.430+0000 7fc2a67fc640 1 -- 172.21.15.47:0/2039509819 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2a8146340 con 0x7fc2a81457f0 2024-03-29T11:20:57.408 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.430+0000 7fc2a67fc640 10 monclient: _finish_auth 0 2024-03-29T11:20:57.408 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.430+0000 7fc2a67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:20:57.408 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.430+0000 7fc2a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:57.409 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.430+0000 7fc2a67fc640 1 -- 172.21.15.47:0/2039509819 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fc2a0001a10 con 0x7fc2a81457f0 2024-03-29T11:20:57.409 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.430+0000 7fc2a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:57.409 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.430+0000 7fc2a67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:57.409 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.430+0000 7fc2a67fc640 10 monclient: dump: 2024-03-29T11:20:57.409 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:20:57.409 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:20:57.409 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:57.409 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:57.409 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:57.409 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:20:57.409 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:20:57.409 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:20:57.409 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:20:57.409 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:20:57.409 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.430+0000 7fc2addfd640 5 monclient: authenticate success, global_id 4550 2024-03-29T11:20:57.410 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.430+0000 7fc2addfd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:57.410 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.433+0000 7fc2a67fc640 1 -- 172.21.15.47:0/2039509819 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc2a0001c70 con 0x7fc2a81457f0 2024-03-29T11:20:57.411 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.433+0000 7fc2a67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:57.411 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.433+0000 7fc2addfd640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:57.411 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.433+0000 7fc2addfd640 4 set_mon_vals no callback set 2024-03-29T11:20:57.411 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.434+0000 7fc2a67fc640 1 -- 172.21.15.47:0/2039509819 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fc2a00020c0 con 0x7fc2a81457f0 2024-03-29T11:20:57.411 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.434+0000 7fc2a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:57.411 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.434+0000 7fc2a67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:57.411 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.434+0000 7fc2a67fc640 10 monclient: dump: 2024-03-29T11:20:57.411 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:20:57.411 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:20:57.411 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:57.411 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:57.411 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:57.411 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:20:57.411 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:20:57.412 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:20:57.412 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:20:57.412 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:20:57.412 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.434+0000 7fc2addfd640 10 monclient: shutdown 2024-03-29T11:20:57.412 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.434+0000 7fc2addfd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:57.412 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.434+0000 7fc2addfd640 1 -- 172.21.15.47:0/2039509819 >> v1:172.21.15.47:6790/0 conn(0x7fc2a81457f0 legacy=0x7fc2a813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:57.412 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.435+0000 7fc2addfd640 1 -- 172.21.15.47:0/2039509819 shutdown_connections 2024-03-29T11:20:57.412 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.435+0000 7fc2addfd640 1 -- 172.21.15.47:0/2039509819 >> 172.21.15.47:0/2039509819 conn(0x7fc2a8066a00 msgr2=0x7fc2a8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:57.413 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.436+0000 7fc2addfd640 1 -- 172.21.15.47:0/2039509819 shutdown_connections 2024-03-29T11:20:57.415 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.437+0000 7fc2addfd640 1 -- 172.21.15.47:0/2039509819 wait complete. 2024-03-29T11:20:57.415 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.437+0000 7fc2addfd640 10 monclient: build_initial_monmap 2024-03-29T11:20:57.415 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.437+0000 7fc2addfd640 10 monclient: monmap: 2024-03-29T11:20:57.415 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:20:57.415 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:57.415 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:20:57.415 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:20:57.415 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:57.415 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:20:57.415 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:20:57.415 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:20:57.415 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:20:57.415 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:20:57.415 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.438+0000 7fc2addfd640 1 librados: starting msgr at 2024-03-29T11:20:57.416 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.438+0000 7fc2addfd640 1 librados: starting objecter 2024-03-29T11:20:57.416 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.438+0000 7fc2addfd640 1 Processor -- start 2024-03-29T11:20:57.416 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.438+0000 7fc2addfd640 1 -- start start 2024-03-29T11:20:57.416 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.438+0000 7fc2addfd640 1 librados: setting wanted keys 2024-03-29T11:20:57.416 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.438+0000 7fc2addfd640 1 librados: calling monclient init 2024-03-29T11:20:57.416 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.438+0000 7fc2addfd640 10 monclient: init 2024-03-29T11:20:57.417 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.439+0000 7fc2addfd640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:57.417 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.439+0000 7fc2addfd640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:20:57.417 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.439+0000 7fc2addfd640 10 monclient(hunting): picked mon.noname-b con 0x7fc2a80a9d70 addr v1:172.21.15.121:6789/0 2024-03-29T11:20:57.417 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.439+0000 7fc2addfd640 10 monclient(hunting): picked mon.noname-a con 0x7fc2a813f650 addr v1:172.21.15.47:6789/0 2024-03-29T11:20:57.417 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.439+0000 7fc2addfd640 10 monclient(hunting): picked mon.noname-c con 0x7fc2a81457f0 addr v1:172.21.15.47:6790/0 2024-03-29T11:20:57.417 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.439+0000 7fc2addfd640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2a81cd8a0 con 0x7fc2a813f650 2024-03-29T11:20:57.418 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.439+0000 7fc2addfd640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2a81cca40 con 0x7fc2a80a9d70 2024-03-29T11:20:57.418 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.439+0000 7fc2addfd640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2a81ccd20 con 0x7fc2a81457f0 2024-03-29T11:20:57.418 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.439+0000 7fc2addfd640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:57.418 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.439+0000 7fc2addfd640 10 monclient(hunting): authenticate will time out at 1300.627531s 2024-03-29T11:20:57.421 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.442+0000 7fc2a77fe640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fc2a80a9d70 0x7fc2a81cdd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.47:44423/0 (socket says 172.21.15.47:44423) 2024-03-29T11:20:57.421 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.442+0000 7fc2a6ffd640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fc2a813f650 0x7fc2a81ce490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:59007/0 (socket says 172.21.15.47:59007) 2024-03-29T11:20:57.421 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.442+0000 7fc2a6ffd640 1 -- 172.21.15.47:0/460103990 learned_addr learned my addr 172.21.15.47:0/460103990 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:20:57.421 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.443+0000 7fc28ffff640 10 client.?.objecter ms_handle_connect 0x7fc2a813f650 2024-03-29T11:20:57.421 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.443+0000 7fc28ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:57.421 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.443+0000 7fc28ffff640 10 client.?.objecter ms_handle_connect 0x7fc2a81457f0 2024-03-29T11:20:57.421 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.443+0000 7fc28ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:57.421 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.443+0000 7fc28ffff640 10 client.?.objecter ms_handle_connect 0x7fc2a80a9d70 2024-03-29T11:20:57.421 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.443+0000 7fc28ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:57.421 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.443+0000 7fc28ffff640 1 -- 172.21.15.47:0/460103990 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 577043272 0 0) 0x7fc2a81ccd20 con 0x7fc2a81457f0 2024-03-29T11:20:57.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.443+0000 7fc28ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:57.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.443+0000 7fc28ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:57.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.443+0000 7fc28ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:57.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.443+0000 7fc28ffff640 10 monclient(hunting): my global_id is 4553 2024-03-29T11:20:57.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.443+0000 7fc28ffff640 1 -- 172.21.15.47:0/460103990 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc294004eb0 con 0x7fc2a81457f0 2024-03-29T11:20:57.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.443+0000 7fc28ffff640 1 -- 172.21.15.47:0/460103990 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4291846361 0 0) 0x7fc2a81cca40 con 0x7fc2a80a9d70 2024-03-29T11:20:57.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.443+0000 7fc28ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:57.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.443+0000 7fc28ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:57.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.443+0000 7fc28ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:57.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.443+0000 7fc28ffff640 10 monclient(hunting): my global_id is 4486 2024-03-29T11:20:57.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.443+0000 7fc28ffff640 1 -- 172.21.15.47:0/460103990 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2a81ccd20 con 0x7fc2a80a9d70 2024-03-29T11:20:57.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.443+0000 7fc28ffff640 1 -- 172.21.15.47:0/460103990 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 432067985 0 0) 0x7fc294004eb0 con 0x7fc2a81457f0 2024-03-29T11:20:57.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.443+0000 7fc28ffff640 1 -- 172.21.15.47:0/460103990 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc2a81cca40 con 0x7fc2a81457f0 2024-03-29T11:20:57.423 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.443+0000 7fc28ffff640 1 -- 172.21.15.47:0/460103990 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fc29c0017f0 con 0x7fc2a81457f0 2024-03-29T11:20:57.423 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.443+0000 7fc28ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:57.423 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.443+0000 7fc28ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:57.423 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.443+0000 7fc28ffff640 10 monclient(hunting): dump: 2024-03-29T11:20:57.423 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:20:57.423 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:20:57.423 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:57.423 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:57.423 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:57.423 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:20:57.423 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:20:57.423 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:20:57.423 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:20:57.423 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:20:57.424 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.443+0000 7fc28ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:57.424 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.443+0000 7fc28ffff640 1 -- 172.21.15.47:0/460103990 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 892065031 0 0) 0x7fc2a81ccd20 con 0x7fc2a80a9d70 2024-03-29T11:20:57.424 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.443+0000 7fc28ffff640 1 -- 172.21.15.47:0/460103990 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc294005920 con 0x7fc2a80a9d70 2024-03-29T11:20:57.424 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.443+0000 7fc28ffff640 1 -- 172.21.15.47:0/460103990 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fc2a0001830 con 0x7fc2a80a9d70 2024-03-29T11:20:57.424 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.443+0000 7fc28ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:57.424 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.443+0000 7fc28ffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:57.424 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.443+0000 7fc28ffff640 10 monclient(hunting): dump: 2024-03-29T11:20:57.424 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:20:57.424 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:20:57.424 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:57.424 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:57.424 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:57.424 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:20:57.424 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:20:57.424 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:20:57.425 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:20:57.425 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:20:57.425 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.444+0000 7fc28ffff640 1 -- 172.21.15.47:0/460103990 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3693906384 0 0) 0x7fc2a81cca40 con 0x7fc2a81457f0 2024-03-29T11:20:57.425 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.444+0000 7fc28ffff640 1 -- 172.21.15.47:0/460103990 >> v1:172.21.15.121:6789/0 conn(0x7fc2a80a9d70 legacy=0x7fc2a81cdd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:57.425 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.444+0000 7fc28ffff640 1 -- 172.21.15.47:0/460103990 >> v1:172.21.15.47:6789/0 conn(0x7fc2a813f650 legacy=0x7fc2a81ce490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:57.425 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.444+0000 7fc28ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:20:57.425 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.445+0000 7fc28ffff640 1 monclient: found mon.c 2024-03-29T11:20:57.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.445+0000 7fc28ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:57.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.445+0000 7fc28ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:20:57.428 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.445+0000 7fc28ffff640 1 -- 172.21.15.47:0/460103990 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2a81cd000 con 0x7fc2a81457f0 2024-03-29T11:20:57.428 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.446+0000 7fc28ffff640 10 monclient: _finish_auth 0 2024-03-29T11:20:57.428 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.446+0000 7fc28ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:20:57.428 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.446+0000 7fc28ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:57.428 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.446+0000 7fc28ffff640 1 -- 172.21.15.47:0/460103990 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc29c001d40 con 0x7fc2a81457f0 2024-03-29T11:20:57.428 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.446+0000 7fc28ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:57.428 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.446+0000 7fc28ffff640 1 -- 172.21.15.47:0/460103990 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fc29c003830 con 0x7fc2a81457f0 2024-03-29T11:20:57.428 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.446+0000 7fc28ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:57.428 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.446+0000 7fc28ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:57.428 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.446+0000 7fc28ffff640 10 monclient: dump: 2024-03-29T11:20:57.428 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:20:57.428 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:20:57.428 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:57.428 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:20:57.429 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:57.429 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:20:57.429 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:20:57.429 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:20:57.429 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:20:57.429 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:20:57.429 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.446+0000 7fc2addfd640 5 monclient: authenticate success, global_id 4553 2024-03-29T11:20:57.429 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.446+0000 7fc2addfd640 10 monclient: _renew_subs 2024-03-29T11:20:57.429 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.446+0000 7fc2addfd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:20:57.429 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.446+0000 7fc2addfd640 1 -- 172.21.15.47:0/460103990 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2a81cbf60 con 0x7fc2a81457f0 2024-03-29T11:20:57.429 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.446+0000 7fc2addfd640 10 client.4553.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:57.429 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.446+0000 7fc2addfd640 10 monclient: _renew_subs 2024-03-29T11:20:57.429 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.446+0000 7fc2addfd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:20:57.429 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.446+0000 7fc2addfd640 1 -- 172.21.15.47:0/460103990 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2a81cc2b0 con 0x7fc2a81457f0 2024-03-29T11:20:57.429 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.446+0000 7fc2addfd640 1 librados: init done 2024-03-29T11:20:57.430 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.447+0000 7fc2a5ffb640 4 set_mon_vals no callback set 2024-03-29T11:20:57.431 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.450+0000 7fc2addfd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:57.431 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.450+0000 7fc28ffff640 1 -- 172.21.15.47:0/460103990 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7fc29c019800 con 0x7fc2a81457f0 2024-03-29T11:20:57.431 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.450+0000 7fc2addfd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:57.431 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.450+0000 7fc2addfd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:20:57.431 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.450+0000 7fc2addfd640 1 -- 172.21.15.47:0/460103990 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc288005080 con 0x7fc2a81457f0 2024-03-29T11:20:57.431 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.450+0000 7fc2a77fe640 10 monclient: get_auth_request con 0x7fc29406e360 auth_method 0 2024-03-29T11:20:57.431 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.450+0000 7fc28ffff640 1 -- 172.21.15.47:0/460103990 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7fc29c0533c0 con 0x7fc2a81457f0 2024-03-29T11:20:57.431 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.450+0000 7fc28ffff640 10 client.4553.objecter ms_dispatch 0x7fc2a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:57.431 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.450+0000 7fc28ffff640 3 client.4553.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:57.432 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.450+0000 7fc28ffff640 3 client.4553.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:57.432 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.450+0000 7fc28ffff640 20 client.4553.objecter dump_active .. 0 homeless 2024-03-29T11:20:57.432 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.453+0000 7fc28ffff640 10 client.4553.objecter ms_handle_connect 0x7fc29406e360 2024-03-29T11:20:57.435 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.459+0000 7fc28ffff640 1 -- 172.21.15.47:0/460103990 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc29c04f660 con 0x7fc2a81457f0 2024-03-29T11:20:57.436 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.459+0000 7fc28ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:57.436 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.459+0000 7fc28ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:57.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.800+0000 7fc2addfd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:57.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.800+0000 7fc2addfd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:57.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.800+0000 7fc2addfd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:20:57.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.800+0000 7fc2addfd640 1 -- 172.21.15.47:0/460103990 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc288005330 con 0x7fc2a81457f0 2024-03-29T11:20:57.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.801+0000 7fc28ffff640 1 -- 172.21.15.47:0/460103990 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7fc29c04f660 con 0x7fc2a81457f0 2024-03-29T11:20:57.778 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.801+0000 7fc28ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:57.778 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.801+0000 7fc28ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:57.778 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:20:57.778 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:57.782 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.804+0000 7fc2addfd640 10 librados: watch_flush enter 2024-03-29T11:20:57.782 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.805+0000 7fc2addfd640 10 librados: watch_flush exit 2024-03-29T11:20:57.782 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.805+0000 7fc2addfd640 20 client.4553.objecter shutdown clearing up homeless session... 2024-03-29T11:20:57.782 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.805+0000 7fc2addfd640 10 client.4553.objecter successfully canceled tick 2024-03-29T11:20:57.782 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.805+0000 7fc2addfd640 1 -- 172.21.15.47:0/460103990 >> v1:172.21.15.121:6816/1964117370 conn(0x7fc29406e360 legacy=0x7fc294070820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:57.782 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.805+0000 7fc2addfd640 10 monclient: shutdown 2024-03-29T11:20:57.782 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.805+0000 7fc2addfd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:57.782 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.805+0000 7fc2addfd640 1 -- 172.21.15.47:0/460103990 >> v1:172.21.15.47:6790/0 conn(0x7fc2a81457f0 legacy=0x7fc2a8140fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:57.784 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.807+0000 7fc2addfd640 1 -- 172.21.15.47:0/460103990 shutdown_connections 2024-03-29T11:20:57.784 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.807+0000 7fc2addfd640 1 -- 172.21.15.47:0/460103990 >> 172.21.15.47:0/460103990 conn(0x7fc2a8066a00 msgr2=0x7fc2a80b5560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:57.788 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.810+0000 7fc2addfd640 1 -- 172.21.15.47:0/460103990 shutdown_connections 2024-03-29T11:20:57.788 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.812+0000 7fc2addfd640 1 -- 172.21.15.47:0/460103990 wait complete. 2024-03-29T11:20:57.788 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:20:57.812+0000 7fc2addfd640 1 librados: shutdown 2024-03-29T11:20:57.805 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:04.612 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:04.764 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.788+0000 7fd7e992d640 10 monclient: get_monmap_and_config 2024-03-29T11:21:04.764 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.788+0000 7fd7e992d640 10 monclient: build_initial_monmap 2024-03-29T11:21:04.765 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.788+0000 7fd7e992d640 10 monclient: monmap: 2024-03-29T11:21:04.765 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:21:04.765 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:21:04.765 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:21:04.788996+0000 2024-03-29T11:21:04.765 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:21:04.788996+0000 2024-03-29T11:21:04.765 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:04.765 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:21:04.765 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:21:04.765 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:21:04.765 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:21:04.765 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:21:04.766 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.789+0000 7fd7e992d640 1 Processor -- start 2024-03-29T11:21:04.766 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.789+0000 7fd7e992d640 1 -- start start 2024-03-29T11:21:04.766 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.789+0000 7fd7e992d640 10 monclient: init 2024-03-29T11:21:04.767 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.790+0000 7fd7e992d640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:04.767 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.790+0000 7fd7e992d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:21:04.767 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.790+0000 7fd7e992d640 10 monclient(hunting): picked mon.noname-c con 0x7fd7dc006a70 addr v1:172.21.15.121:6789/0 2024-03-29T11:21:04.767 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.790+0000 7fd7e992d640 10 monclient(hunting): picked mon.noname-b con 0x7fd7dc02b100 addr v1:172.21.15.47:6790/0 2024-03-29T11:21:04.767 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.790+0000 7fd7e992d640 10 monclient(hunting): picked mon.noname-a con 0x7fd7dc029c30 addr v1:172.21.15.47:6789/0 2024-03-29T11:21:04.767 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.790+0000 7fd7e992d640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7dc02a270 con 0x7fd7dc029c30 2024-03-29T11:21:04.767 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.790+0000 7fd7e992d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7dc02c440 con 0x7fd7dc006a70 2024-03-29T11:21:04.767 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.790+0000 7fd7e992d640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7dc0c9440 con 0x7fd7dc02b100 2024-03-29T11:21:04.767 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.790+0000 7fd7e992d640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:04.767 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.790+0000 7fd7e992d640 10 monclient(hunting): authenticate will time out at 1307.978305s 2024-03-29T11:21:04.770 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.793+0000 7fd7e892b640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fd7dc006a70 0x7fd7dc0c8d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.47:49377/0 (socket says 172.21.15.47:49377) 2024-03-29T11:21:04.771 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.794+0000 7fd7e892b640 1 -- 172.21.15.47:0/1522718798 learned_addr learned my addr 172.21.15.47:0/1522718798 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:21:04.781 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.800+0000 7fd7e37fe640 1 -- 172.21.15.47:0/1522718798 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 745008022 0 0) 0x7fd7dc02c440 con 0x7fd7dc006a70 2024-03-29T11:21:04.781 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.800+0000 7fd7e37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:04.781 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.800+0000 7fd7e37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:04.782 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.800+0000 7fd7e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:04.782 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.800+0000 7fd7e37fe640 10 monclient(hunting): my global_id is 4489 2024-03-29T11:21:04.782 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.800+0000 7fd7e37fe640 1 -- 172.21.15.47:0/1522718798 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7d4003520 con 0x7fd7dc006a70 2024-03-29T11:21:04.782 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.800+0000 7fd7e37fe640 1 -- 172.21.15.47:0/1522718798 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1774197336 0 0) 0x7fd7dc0c9440 con 0x7fd7dc02b100 2024-03-29T11:21:04.782 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.800+0000 7fd7e37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:04.782 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.800+0000 7fd7e37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:04.782 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.800+0000 7fd7e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:04.782 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.800+0000 7fd7e37fe640 10 monclient(hunting): my global_id is 4556 2024-03-29T11:21:04.782 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.800+0000 7fd7e37fe640 1 -- 172.21.15.47:0/1522718798 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7dc02c440 con 0x7fd7dc02b100 2024-03-29T11:21:04.782 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.800+0000 7fd7e37fe640 1 -- 172.21.15.47:0/1522718798 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1500386421 0 0) 0x7fd7dc02a270 con 0x7fd7dc029c30 2024-03-29T11:21:04.782 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.800+0000 7fd7e37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:04.783 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.800+0000 7fd7e37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:04.783 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.800+0000 7fd7e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:04.783 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.800+0000 7fd7e37fe640 10 monclient(hunting): my global_id is 4497 2024-03-29T11:21:04.783 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.800+0000 7fd7e37fe640 1 -- 172.21.15.47:0/1522718798 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7dc0c9440 con 0x7fd7dc029c30 2024-03-29T11:21:04.783 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.800+0000 7fd7e37fe640 1 -- 172.21.15.47:0/1522718798 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 855359638 0 0) 0x7fd7dc02c440 con 0x7fd7dc02b100 2024-03-29T11:21:04.783 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.800+0000 7fd7e37fe640 1 -- 172.21.15.47:0/1522718798 >> v1:172.21.15.121:6789/0 conn(0x7fd7dc006a70 legacy=0x7fd7dc0c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:04.783 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.800+0000 7fd7e37fe640 1 -- 172.21.15.47:0/1522718798 >> v1:172.21.15.47:6789/0 conn(0x7fd7dc029c30 legacy=0x7fd7dc0d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:04.783 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.800+0000 7fd7e37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:21:04.784 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.800+0000 7fd7e37fe640 1 monclient: found mon.noname-b 2024-03-29T11:21:04.784 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.800+0000 7fd7e37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:04.784 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.800+0000 7fd7e37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-29T11:21:04.784 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.800+0000 7fd7e37fe640 1 -- 172.21.15.47:0/1522718798 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7dc0c95d0 con 0x7fd7dc02b100 2024-03-29T11:21:04.784 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.800+0000 7fd7e37fe640 10 monclient: _finish_auth 0 2024-03-29T11:21:04.784 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.800+0000 7fd7e37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:21:04.784 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.800+0000 7fd7e37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:04.784 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.800+0000 7fd7e992d640 5 monclient: authenticate success, global_id 4556 2024-03-29T11:21:04.784 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.800+0000 7fd7e992d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:04.784 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.800+0000 7fd7e37fe640 1 -- 172.21.15.47:0/1522718798 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fd7d0001520 con 0x7fd7dc02b100 2024-03-29T11:21:04.784 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.800+0000 7fd7e37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:04.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.800+0000 7fd7e37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:04.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.800+0000 7fd7e37fe640 10 monclient: dump: 2024-03-29T11:21:04.785 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:21:04.785 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:21:04.785 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:04.785 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:04.785 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:04.785 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:21:04.785 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:21:04.785 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:21:04.785 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:21:04.785 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:21:04.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.800+0000 7fd7e992d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:04.786 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.801+0000 7fd7e37fe640 1 -- 172.21.15.47:0/1522718798 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd7d0001740 con 0x7fd7dc02b100 2024-03-29T11:21:04.786 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.801+0000 7fd7e37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:04.786 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.801+0000 7fd7e37fe640 1 -- 172.21.15.47:0/1522718798 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fd7d0001ad0 con 0x7fd7dc02b100 2024-03-29T11:21:04.786 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.801+0000 7fd7e37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:04.786 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.801+0000 7fd7e37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:04.786 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.801+0000 7fd7e37fe640 10 monclient: dump: 2024-03-29T11:21:04.786 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:21:04.786 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:21:04.786 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:04.786 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:04.787 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:04.787 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:21:04.787 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:21:04.787 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:21:04.787 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:21:04.787 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:21:04.787 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.801+0000 7fd7e992d640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:04.787 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.801+0000 7fd7e992d640 4 set_mon_vals no callback set 2024-03-29T11:21:04.787 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.802+0000 7fd7e992d640 10 monclient: shutdown 2024-03-29T11:21:04.787 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.802+0000 7fd7e992d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:04.787 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.802+0000 7fd7e992d640 1 -- 172.21.15.47:0/1522718798 >> v1:172.21.15.47:6790/0 conn(0x7fd7dc02b100 legacy=0x7fd7dc0cf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:04.787 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.802+0000 7fd7e992d640 1 -- 172.21.15.47:0/1522718798 shutdown_connections 2024-03-29T11:21:04.788 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.802+0000 7fd7e992d640 1 -- 172.21.15.47:0/1522718798 >> 172.21.15.47:0/1522718798 conn(0x7fd7dc0058f0 msgr2=0x7fd7dc005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:04.788 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.805+0000 7fd7e992d640 1 -- 172.21.15.47:0/1522718798 shutdown_connections 2024-03-29T11:21:04.790 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.805+0000 7fd7e992d640 1 -- 172.21.15.47:0/1522718798 wait complete. 2024-03-29T11:21:04.790 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.806+0000 7fd7e992d640 10 monclient: build_initial_monmap 2024-03-29T11:21:04.790 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.807+0000 7fd7e992d640 10 monclient: monmap: 2024-03-29T11:21:04.790 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:21:04.790 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:04.790 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:21:04.791 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:21:04.791 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:04.791 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:21:04.791 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:21:04.791 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:21:04.791 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:21:04.791 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:21:04.791 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.807+0000 7fd7e992d640 1 librados: starting msgr at 2024-03-29T11:21:04.791 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.807+0000 7fd7e992d640 1 librados: starting objecter 2024-03-29T11:21:04.791 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.807+0000 7fd7e992d640 1 Processor -- start 2024-03-29T11:21:04.791 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.807+0000 7fd7e992d640 1 -- start start 2024-03-29T11:21:04.791 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.807+0000 7fd7e992d640 1 librados: setting wanted keys 2024-03-29T11:21:04.792 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.808+0000 7fd7e992d640 1 librados: calling monclient init 2024-03-29T11:21:04.792 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.808+0000 7fd7e992d640 10 monclient: init 2024-03-29T11:21:04.792 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.809+0000 7fd7e992d640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:04.792 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.809+0000 7fd7e992d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:21:04.792 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.809+0000 7fd7e992d640 10 monclient(hunting): picked mon.noname-a con 0x7fd7dc006a70 addr v1:172.21.15.47:6789/0 2024-03-29T11:21:04.792 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.813+0000 7fd7e992d640 10 monclient(hunting): picked mon.noname-c con 0x7fd7dc029c30 addr v1:172.21.15.47:6790/0 2024-03-29T11:21:04.792 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.813+0000 7fd7e3fff640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fd7dc029c30 0x7fd7dc161320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:54013/0 (socket says 172.21.15.47:54013) 2024-03-29T11:21:04.792 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.813+0000 7fd7e3fff640 1 -- 172.21.15.47:0/900161791 learned_addr learned my addr 172.21.15.47:0/900161791 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:21:04.792 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.813+0000 7fd7e992d640 10 monclient(hunting): picked mon.noname-b con 0x7fd7dc0ccd10 addr v1:172.21.15.121:6789/0 2024-03-29T11:21:04.792 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.813+0000 7fd7e17fa640 10 client.?.objecter ms_handle_connect 0x7fd7dc029c30 2024-03-29T11:21:04.792 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.813+0000 7fd7e17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:04.792 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.813+0000 7fd7e992d640 1 -- 172.21.15.47:0/900161791 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7dc02d810 con 0x7fd7dc006a70 2024-03-29T11:21:04.794 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.815+0000 7fd7e992d640 1 -- 172.21.15.47:0/900161791 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7dc160bb0 con 0x7fd7dc0ccd10 2024-03-29T11:21:04.794 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.815+0000 7fd7e992d640 1 -- 172.21.15.47:0/900161791 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7dc161a30 con 0x7fd7dc029c30 2024-03-29T11:21:04.795 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.815+0000 7fd7e992d640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:04.795 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.815+0000 7fd7e992d640 10 monclient(hunting): authenticate will time out at 1308.003122s 2024-03-29T11:21:04.795 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.816+0000 7fd7e17fa640 10 client.?.objecter ms_handle_connect 0x7fd7dc006a70 2024-03-29T11:21:04.795 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.816+0000 7fd7e17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:04.795 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.816+0000 7fd7e17fa640 1 -- 172.21.15.47:0/900161791 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3892490382 0 0) 0x7fd7dc02d810 con 0x7fd7dc006a70 2024-03-29T11:21:04.795 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.816+0000 7fd7e17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:21:04.795 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.816+0000 7fd7e17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:04.795 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.816+0000 7fd7e17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:04.795 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.816+0000 7fd7e17fa640 10 monclient(hunting): my global_id is 4500 2024-03-29T11:21:04.795 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.816+0000 7fd7e17fa640 1 -- 172.21.15.47:0/900161791 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7c8004eb0 con 0x7fd7dc006a70 2024-03-29T11:21:04.795 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.817+0000 7fd7e17fa640 1 -- 172.21.15.47:0/900161791 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2228171484 0 0) 0x7fd7dc161a30 con 0x7fd7dc029c30 2024-03-29T11:21:04.795 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.817+0000 7fd7e17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:21:04.796 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.817+0000 7fd7e17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:04.796 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.817+0000 7fd7e17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:04.796 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.817+0000 7fd7e17fa640 10 monclient(hunting): my global_id is 4559 2024-03-29T11:21:04.796 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.817+0000 7fd7e17fa640 1 -- 172.21.15.47:0/900161791 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7dc02d810 con 0x7fd7dc029c30 2024-03-29T11:21:04.796 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.818+0000 7fd7e17fa640 1 -- 172.21.15.47:0/900161791 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 761907652 0 0) 0x7fd7c8004eb0 con 0x7fd7dc006a70 2024-03-29T11:21:04.796 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.818+0000 7fd7e17fa640 1 -- 172.21.15.47:0/900161791 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd7c8005490 con 0x7fd7dc006a70 2024-03-29T11:21:04.796 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.818+0000 7fd7e17fa640 1 -- 172.21.15.47:0/900161791 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fd7d8001950 con 0x7fd7dc006a70 2024-03-29T11:21:04.796 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.818+0000 7fd7e17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:04.796 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.818+0000 7fd7e17fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:04.796 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.818+0000 7fd7e17fa640 10 monclient(hunting): dump: 2024-03-29T11:21:04.796 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:21:04.796 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:21:04.797 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:04.797 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:04.797 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:04.797 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:21:04.797 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:21:04.797 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:21:04.797 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:21:04.797 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:21:04.797 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.818+0000 7fd7e17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:04.797 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.818+0000 7fd7e17fa640 1 -- 172.21.15.47:0/900161791 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2376549432 0 0) 0x7fd7dc02d810 con 0x7fd7dc029c30 2024-03-29T11:21:04.798 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.818+0000 7fd7e17fa640 1 -- 172.21.15.47:0/900161791 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd7c8004eb0 con 0x7fd7dc029c30 2024-03-29T11:21:04.798 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.818+0000 7fd7e17fa640 1 -- 172.21.15.47:0/900161791 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fd7d0001b50 con 0x7fd7dc029c30 2024-03-29T11:21:04.798 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.818+0000 7fd7e17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:04.798 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.818+0000 7fd7e17fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:04.798 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.818+0000 7fd7e17fa640 10 monclient(hunting): dump: 2024-03-29T11:21:04.798 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:21:04.798 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:21:04.798 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:04.799 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:04.799 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:04.799 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:21:04.799 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:21:04.799 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:21:04.799 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:21:04.799 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:21:04.799 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.818+0000 7fd7e17fa640 1 -- 172.21.15.47:0/900161791 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1291876639 0 0) 0x7fd7c8005490 con 0x7fd7dc006a70 2024-03-29T11:21:04.799 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.818+0000 7fd7e17fa640 1 -- 172.21.15.47:0/900161791 >> v1:172.21.15.47:6790/0 conn(0x7fd7dc029c30 legacy=0x7fd7dc161320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:04.799 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.818+0000 7fd7e17fa640 1 -- 172.21.15.47:0/900161791 >> v1:172.21.15.121:6789/0 conn(0x7fd7dc0ccd10 legacy=0x7fd7dc031c60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:21:04.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.819+0000 7fd7e17fa640 10 monclient: _finish_hunting 0 2024-03-29T11:21:04.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.819+0000 7fd7e17fa640 1 monclient: found mon.a 2024-03-29T11:21:04.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.819+0000 7fd7e17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:04.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.819+0000 7fd7e17fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:21:04.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.819+0000 7fd7e17fa640 1 -- 172.21.15.47:0/900161791 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7dc160e00 con 0x7fd7dc006a70 2024-03-29T11:21:04.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.819+0000 7fd7e17fa640 10 monclient: _finish_auth 0 2024-03-29T11:21:04.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.819+0000 7fd7e17fa640 10 monclient: _check_auth_tickets 2024-03-29T11:21:04.802 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.819+0000 7fd7e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:04.802 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.819+0000 7fd7e992d640 5 monclient: authenticate success, global_id 4500 2024-03-29T11:21:04.802 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.819+0000 7fd7e992d640 10 monclient: _renew_subs 2024-03-29T11:21:04.802 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.819+0000 7fd7e992d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:21:04.802 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.819+0000 7fd7e992d640 1 -- 172.21.15.47:0/900161791 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7dc032370 con 0x7fd7dc006a70 2024-03-29T11:21:04.802 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.819+0000 7fd7e992d640 10 client.4500.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:04.802 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.819+0000 7fd7e992d640 10 monclient: _renew_subs 2024-03-29T11:21:04.802 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.819+0000 7fd7e992d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:21:04.802 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.819+0000 7fd7e992d640 1 -- 172.21.15.47:0/900161791 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7dc02dea0 con 0x7fd7dc006a70 2024-03-29T11:21:04.802 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.819+0000 7fd7e992d640 1 librados: init done 2024-03-29T11:21:04.802 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.820+0000 7fd7e992d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:04.803 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.820+0000 7fd7e992d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:04.803 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.820+0000 7fd7e992d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:21:04.803 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.820+0000 7fd7e992d640 1 -- 172.21.15.47:0/900161791 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7dc004cd0 con 0x7fd7dc006a70 2024-03-29T11:21:04.803 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.823+0000 7fd7e17fa640 1 -- 172.21.15.47:0/900161791 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd7d80047e0 con 0x7fd7dc006a70 2024-03-29T11:21:04.803 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.823+0000 7fd7e17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:04.803 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.823+0000 7fd7e17fa640 1 -- 172.21.15.47:0/900161791 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fd7d8004b30 con 0x7fd7dc006a70 2024-03-29T11:21:04.803 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.823+0000 7fd7e17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:04.803 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.823+0000 7fd7e17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:04.803 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.823+0000 7fd7e17fa640 10 monclient: dump: 2024-03-29T11:21:04.803 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:21:04.803 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:21:04.803 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:04.804 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:04.804 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:04.804 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:21:04.804 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:21:04.804 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:21:04.804 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:21:04.804 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:21:04.804 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.823+0000 7fd7e2ffd640 4 set_mon_vals no callback set 2024-03-29T11:21:04.805 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.825+0000 7fd7e17fa640 1 -- 172.21.15.47:0/900161791 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7fd7d801ab00 con 0x7fd7dc006a70 2024-03-29T11:21:04.805 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.825+0000 7fd7e17fa640 1 -- 172.21.15.47:0/900161791 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7fd7d8084a00 con 0x7fd7dc006a70 2024-03-29T11:21:04.805 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.825+0000 7fd7e17fa640 10 client.4500.objecter ms_dispatch 0x7fd7e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:04.805 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.825+0000 7fd7e17fa640 3 client.4500.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:04.806 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.825+0000 7fd7e17fa640 3 client.4500.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:04.806 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.826+0000 7fd7e17fa640 20 client.4500.objecter dump_active .. 0 homeless 2024-03-29T11:21:04.806 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.826+0000 7fd7e3fff640 10 monclient: get_auth_request con 0x7fd7c806e250 auth_method 0 2024-03-29T11:21:04.806 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.828+0000 7fd7e17fa640 10 client.4500.objecter ms_handle_connect 0x7fd7c806e250 2024-03-29T11:21:04.809 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.833+0000 7fd7e17fa640 1 -- 172.21.15.47:0/900161791 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd7d8050340 con 0x7fd7dc006a70 2024-03-29T11:21:04.809 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.833+0000 7fd7e17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:04.809 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:04.833+0000 7fd7e17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:05.130 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:05.153+0000 7fd7e992d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:05.130 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:05.153+0000 7fd7e992d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:05.130 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:05.153+0000 7fd7e992d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:21:05.130 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:05.153+0000 7fd7e992d640 1 -- 172.21.15.47:0/900161791 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd7dc160f90 con 0x7fd7dc006a70 2024-03-29T11:21:05.130 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:05.154+0000 7fd7e17fa640 1 -- 172.21.15.47:0/900161791 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7fd7d8052270 con 0x7fd7dc006a70 2024-03-29T11:21:05.131 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:05.154+0000 7fd7e17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:05.131 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:05.154+0000 7fd7e17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:05.134 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:21:05.134 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:05.137 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:05.160+0000 7fd7e992d640 10 librados: watch_flush enter 2024-03-29T11:21:05.138 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:05.161+0000 7fd7e992d640 10 librados: watch_flush exit 2024-03-29T11:21:05.138 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:05.161+0000 7fd7e992d640 20 client.4500.objecter shutdown clearing up homeless session... 2024-03-29T11:21:05.138 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:05.161+0000 7fd7e992d640 10 client.4500.objecter successfully canceled tick 2024-03-29T11:21:05.138 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:05.161+0000 7fd7e992d640 1 -- 172.21.15.47:0/900161791 >> v1:172.21.15.121:6816/1964117370 conn(0x7fd7c806e250 legacy=0x7fd7c8070710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:05.138 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:05.161+0000 7fd7e992d640 10 monclient: shutdown 2024-03-29T11:21:05.138 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:05.161+0000 7fd7e992d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:05.138 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:05.161+0000 7fd7e992d640 1 -- 172.21.15.47:0/900161791 >> v1:172.21.15.47:6789/0 conn(0x7fd7dc006a70 legacy=0x7fd7dc02d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:05.139 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:05.162+0000 7fd7e992d640 1 -- 172.21.15.47:0/900161791 shutdown_connections 2024-03-29T11:21:05.139 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:05.162+0000 7fd7e992d640 1 -- 172.21.15.47:0/900161791 >> 172.21.15.47:0/900161791 conn(0x7fd7dc0058f0 msgr2=0x7fd7dc0c3e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:05.139 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:05.162+0000 7fd7e992d640 1 -- 172.21.15.47:0/900161791 shutdown_connections 2024-03-29T11:21:05.142 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:05.163+0000 7fd7e992d640 1 -- 172.21.15.47:0/900161791 wait complete. 2024-03-29T11:21:05.142 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:05.163+0000 7fd7e992d640 1 librados: shutdown 2024-03-29T11:21:05.161 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:11.968 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:12.144 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.167+0000 7f6f39e0d640 10 monclient: get_monmap_and_config 2024-03-29T11:21:12.144 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.167+0000 7f6f39e0d640 10 monclient: build_initial_monmap 2024-03-29T11:21:12.144 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.167+0000 7f6f39e0d640 10 monclient: monmap: 2024-03-29T11:21:12.144 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:21:12.144 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:21:12.144 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:21:12.168587+0000 2024-03-29T11:21:12.144 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:21:12.168587+0000 2024-03-29T11:21:12.144 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:12.144 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:21:12.145 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:21:12.145 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:21:12.145 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:21:12.145 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:21:12.145 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.168+0000 7f6f39e0d640 1 Processor -- start 2024-03-29T11:21:12.149 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.168+0000 7f6f39e0d640 1 -- start start 2024-03-29T11:21:12.149 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.168+0000 7f6f39e0d640 10 monclient: init 2024-03-29T11:21:12.149 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.169+0000 7f6f39e0d640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:12.149 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.169+0000 7f6f39e0d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:21:12.149 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.172+0000 7f6f39e0d640 10 monclient(hunting): picked mon.noname-b con 0x7f6f340b0770 addr v1:172.21.15.47:6790/0 2024-03-29T11:21:12.149 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.172+0000 7f6f337fe640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f6f340b0770 0x7f6f34149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:60053/0 (socket says 172.21.15.47:60053) 2024-03-29T11:21:12.152 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.172+0000 7f6f39e0d640 10 monclient(hunting): picked mon.noname-c con 0x7f6f3414a700 addr v1:172.21.15.121:6789/0 2024-03-29T11:21:12.152 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.175+0000 7f6f39e0d640 10 monclient(hunting): picked mon.noname-a con 0x7f6f340ac5a0 addr v1:172.21.15.47:6789/0 2024-03-29T11:21:12.153 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.175+0000 7f6f337fe640 1 -- 172.21.15.47:0/3314570406 learned_addr learned my addr 172.21.15.47:0/3314570406 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:21:12.153 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.175+0000 7f6f39e0d640 1 -- 172.21.15.47:0/3314570406 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f340aba70 con 0x7f6f340ac5a0 2024-03-29T11:21:12.153 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.176+0000 7f6f39e0d640 1 -- 172.21.15.47:0/3314570406 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f340aecb0 con 0x7f6f3414a700 2024-03-29T11:21:12.153 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.176+0000 7f6f39e0d640 1 -- 172.21.15.47:0/3314570406 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f340af150 con 0x7f6f340b0770 2024-03-29T11:21:12.153 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.176+0000 7f6f39e0d640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:12.153 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.176+0000 7f6f39e0d640 10 monclient(hunting): authenticate will time out at 1315.363977s 2024-03-29T11:21:12.153 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.176+0000 7f6f327fc640 1 -- 172.21.15.47:0/3314570406 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3341209994 0 0) 0x7f6f340aecb0 con 0x7f6f3414a700 2024-03-29T11:21:12.153 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.176+0000 7f6f327fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:12.154 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.176+0000 7f6f327fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:12.154 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.176+0000 7f6f327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:12.154 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.176+0000 7f6f327fc640 10 monclient(hunting): my global_id is 4492 2024-03-29T11:21:12.154 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.176+0000 7f6f327fc640 1 -- 172.21.15.47:0/3314570406 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f1c003520 con 0x7f6f3414a700 2024-03-29T11:21:12.154 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.177+0000 7f6f327fc640 1 -- 172.21.15.47:0/3314570406 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3803942760 0 0) 0x7f6f340af150 con 0x7f6f340b0770 2024-03-29T11:21:12.154 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.177+0000 7f6f327fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:12.154 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.177+0000 7f6f327fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:12.154 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.177+0000 7f6f327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:12.155 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.177+0000 7f6f327fc640 10 monclient(hunting): my global_id is 4562 2024-03-29T11:21:12.155 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.177+0000 7f6f327fc640 1 -- 172.21.15.47:0/3314570406 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f340aecb0 con 0x7f6f340b0770 2024-03-29T11:21:12.157 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.181+0000 7f6f327fc640 1 -- 172.21.15.47:0/3314570406 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2810726382 0 0) 0x7f6f340aba70 con 0x7f6f340ac5a0 2024-03-29T11:21:12.157 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.181+0000 7f6f327fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:12.158 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.181+0000 7f6f327fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:12.158 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.181+0000 7f6f327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:12.158 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.181+0000 7f6f327fc640 10 monclient(hunting): my global_id is 4503 2024-03-29T11:21:12.158 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.181+0000 7f6f327fc640 1 -- 172.21.15.47:0/3314570406 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f340af150 con 0x7f6f340ac5a0 2024-03-29T11:21:12.158 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.181+0000 7f6f327fc640 1 -- 172.21.15.47:0/3314570406 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2091523501 0 0) 0x7f6f340aecb0 con 0x7f6f340b0770 2024-03-29T11:21:12.158 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.181+0000 7f6f327fc640 1 -- 172.21.15.47:0/3314570406 >> v1:172.21.15.121:6789/0 conn(0x7f6f3414a700 legacy=0x7f6f340abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:12.158 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.181+0000 7f6f327fc640 1 -- 172.21.15.47:0/3314570406 >> v1:172.21.15.47:6789/0 conn(0x7f6f340ac5a0 legacy=0x7f6f34151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:12.158 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.181+0000 7f6f327fc640 10 monclient: _finish_hunting 0 2024-03-29T11:21:12.158 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.181+0000 7f6f327fc640 1 monclient: found mon.noname-b 2024-03-29T11:21:12.158 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.181+0000 7f6f327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:12.158 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.181+0000 7f6f327fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-29T11:21:12.158 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.181+0000 7f6f327fc640 1 -- 172.21.15.47:0/3314570406 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f340ac9d0 con 0x7f6f340b0770 2024-03-29T11:21:12.158 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.181+0000 7f6f327fc640 10 monclient: _finish_auth 0 2024-03-29T11:21:12.158 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.181+0000 7f6f327fc640 10 monclient: _check_auth_tickets 2024-03-29T11:21:12.159 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.181+0000 7f6f327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:12.159 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.181+0000 7f6f327fc640 1 -- 172.21.15.47:0/3314570406 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f6f240015f0 con 0x7f6f340b0770 2024-03-29T11:21:12.159 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.181+0000 7f6f327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:12.159 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.181+0000 7f6f327fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:12.159 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.181+0000 7f6f327fc640 10 monclient: dump: 2024-03-29T11:21:12.159 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:21:12.159 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:21:12.159 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:12.159 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:12.159 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:12.159 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:21:12.159 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:21:12.159 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:21:12.159 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:21:12.159 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:21:12.159 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.181+0000 7f6f327fc640 1 -- 172.21.15.47:0/3314570406 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6f24001750 con 0x7f6f340b0770 2024-03-29T11:21:12.160 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.181+0000 7f6f327fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:12.160 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.181+0000 7f6f327fc640 1 -- 172.21.15.47:0/3314570406 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f6f24001ba0 con 0x7f6f340b0770 2024-03-29T11:21:12.160 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.181+0000 7f6f327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:12.160 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.181+0000 7f6f327fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:12.160 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.181+0000 7f6f327fc640 10 monclient: dump: 2024-03-29T11:21:12.160 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:21:12.160 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:21:12.160 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:12.160 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:12.160 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:12.160 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:21:12.160 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:21:12.160 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:21:12.160 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:21:12.160 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:21:12.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.183+0000 7f6f39e0d640 5 monclient: authenticate success, global_id 4562 2024-03-29T11:21:12.162 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.183+0000 7f6f39e0d640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:12.162 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.183+0000 7f6f39e0d640 4 set_mon_vals no callback set 2024-03-29T11:21:12.162 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.183+0000 7f6f39e0d640 10 monclient: shutdown 2024-03-29T11:21:12.162 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.183+0000 7f6f39e0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:12.162 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.183+0000 7f6f39e0d640 1 -- 172.21.15.47:0/3314570406 >> v1:172.21.15.47:6790/0 conn(0x7f6f340b0770 legacy=0x7f6f34149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:12.162 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.185+0000 7f6f39e0d640 1 -- 172.21.15.47:0/3314570406 shutdown_connections 2024-03-29T11:21:12.162 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.185+0000 7f6f39e0d640 1 -- 172.21.15.47:0/3314570406 >> 172.21.15.47:0/3314570406 conn(0x7f6f3406b660 msgr2=0x7f6f3406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:12.162 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.185+0000 7f6f39e0d640 1 -- 172.21.15.47:0/3314570406 shutdown_connections 2024-03-29T11:21:12.162 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.185+0000 7f6f39e0d640 1 -- 172.21.15.47:0/3314570406 wait complete. 2024-03-29T11:21:12.163 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.186+0000 7f6f39e0d640 10 monclient: build_initial_monmap 2024-03-29T11:21:12.163 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.186+0000 7f6f39e0d640 10 monclient: monmap: 2024-03-29T11:21:12.164 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:21:12.164 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:12.164 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:21:12.164 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:21:12.164 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:12.164 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:21:12.164 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:21:12.164 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:21:12.164 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:21:12.164 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:21:12.164 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.186+0000 7f6f39e0d640 1 librados: starting msgr at 2024-03-29T11:21:12.164 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.186+0000 7f6f39e0d640 1 librados: starting objecter 2024-03-29T11:21:12.164 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.186+0000 7f6f39e0d640 1 Processor -- start 2024-03-29T11:21:12.164 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.187+0000 7f6f39e0d640 1 -- start start 2024-03-29T11:21:12.164 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.187+0000 7f6f39e0d640 1 librados: setting wanted keys 2024-03-29T11:21:12.164 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.187+0000 7f6f39e0d640 1 librados: calling monclient init 2024-03-29T11:21:12.165 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.187+0000 7f6f39e0d640 10 monclient: init 2024-03-29T11:21:12.165 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.187+0000 7f6f39e0d640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:12.165 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.187+0000 7f6f39e0d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:21:12.166 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.190+0000 7f6f39e0d640 10 monclient(hunting): picked mon.noname-c con 0x7f6f340ac5a0 addr v1:172.21.15.47:6790/0 2024-03-29T11:21:12.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.191+0000 7f6f39e0d640 10 monclient(hunting): picked mon.noname-a con 0x7f6f340b0770 addr v1:172.21.15.47:6789/0 2024-03-29T11:21:12.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.191+0000 7f6f39e0d640 10 monclient(hunting): picked mon.noname-b con 0x7f6f3414a700 addr v1:172.21.15.121:6789/0 2024-03-29T11:21:12.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.191+0000 7f6f39e0d640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f341274c0 con 0x7f6f340b0770 2024-03-29T11:21:12.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.191+0000 7f6f39e0d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f34128530 con 0x7f6f3414a700 2024-03-29T11:21:12.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.191+0000 7f6f39e0d640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f341db6d0 con 0x7f6f340ac5a0 2024-03-29T11:21:12.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.191+0000 7f6f39e0d640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:12.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.191+0000 7f6f39e0d640 10 monclient(hunting): authenticate will time out at 1315.379283s 2024-03-29T11:21:12.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.191+0000 7f6f337fe640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f6f340ac5a0 0x7f6f34126db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:60143/0 (socket says 172.21.15.47:60143) 2024-03-29T11:21:12.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.191+0000 7f6f337fe640 1 -- 172.21.15.47:0/112831919 learned_addr learned my addr 172.21.15.47:0/112831919 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:21:12.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.191+0000 7f6f1bfff640 10 client.?.objecter ms_handle_connect 0x7f6f340ac5a0 2024-03-29T11:21:12.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.191+0000 7f6f1bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:12.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.192+0000 7f6f1bfff640 1 -- 172.21.15.47:0/112831919 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3099384197 0 0) 0x7f6f341db6d0 con 0x7f6f340ac5a0 2024-03-29T11:21:12.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.192+0000 7f6f1bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:21:12.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.192+0000 7f6f1bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:12.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.192+0000 7f6f1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:12.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.192+0000 7f6f1bfff640 10 monclient(hunting): my global_id is 4565 2024-03-29T11:21:12.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.192+0000 7f6f1bfff640 1 -- 172.21.15.47:0/112831919 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f20004eb0 con 0x7f6f340ac5a0 2024-03-29T11:21:12.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.192+0000 7f6f1bfff640 1 -- 172.21.15.47:0/112831919 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4078158152 0 0) 0x7f6f20004eb0 con 0x7f6f340ac5a0 2024-03-29T11:21:12.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.192+0000 7f6f1bfff640 1 -- 172.21.15.47:0/112831919 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6f341db6d0 con 0x7f6f340ac5a0 2024-03-29T11:21:12.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.192+0000 7f6f1bfff640 1 -- 172.21.15.47:0/112831919 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f6f24001b60 con 0x7f6f340ac5a0 2024-03-29T11:21:12.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.192+0000 7f6f1bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:12.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.192+0000 7f6f1bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:12.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.192+0000 7f6f1bfff640 10 monclient(hunting): dump: 2024-03-29T11:21:12.171 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:21:12.172 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:21:12.172 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:12.172 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:12.172 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:12.172 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:21:12.172 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:21:12.172 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:21:12.172 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:21:12.172 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:21:12.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.192+0000 7f6f1bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:12.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.193+0000 7f6f1bfff640 1 -- 172.21.15.47:0/112831919 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2387118395 0 0) 0x7f6f341db6d0 con 0x7f6f340ac5a0 2024-03-29T11:21:12.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.193+0000 7f6f1bfff640 1 -- 172.21.15.47:0/112831919 >> v1:172.21.15.121:6789/0 conn(0x7f6f3414a700 legacy=0x7f6f34145160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:21:12.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.195+0000 7f6f1bfff640 1 -- 172.21.15.47:0/112831919 >> v1:172.21.15.47:6789/0 conn(0x7f6f340b0770 legacy=0x7f6f341dbbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:12.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.195+0000 7f6f1bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:21:12.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.195+0000 7f6f1bfff640 1 monclient: found mon.c 2024-03-29T11:21:12.175 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.195+0000 7f6f1bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:12.175 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.195+0000 7f6f1bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:21:12.175 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.195+0000 7f6f1bfff640 1 -- 172.21.15.47:0/112831919 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f341239d0 con 0x7f6f340ac5a0 2024-03-29T11:21:12.175 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.195+0000 7f6f1bfff640 10 monclient: _finish_auth 0 2024-03-29T11:21:12.175 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.195+0000 7f6f1bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:21:12.175 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.195+0000 7f6f1bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:12.175 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.196+0000 7f6f39e0d640 5 monclient: authenticate success, global_id 4565 2024-03-29T11:21:12.175 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.196+0000 7f6f39e0d640 10 monclient: _renew_subs 2024-03-29T11:21:12.175 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.196+0000 7f6f39e0d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:21:12.175 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.196+0000 7f6f39e0d640 1 -- 172.21.15.47:0/112831919 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f341dc300 con 0x7f6f340ac5a0 2024-03-29T11:21:12.175 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.196+0000 7f6f39e0d640 10 client.4565.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:12.175 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.196+0000 7f6f39e0d640 10 monclient: _renew_subs 2024-03-29T11:21:12.175 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.197+0000 7f6f39e0d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:21:12.175 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.197+0000 7f6f39e0d640 1 -- 172.21.15.47:0/112831919 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f34145a50 con 0x7f6f340ac5a0 2024-03-29T11:21:12.175 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.197+0000 7f6f39e0d640 1 librados: init done 2024-03-29T11:21:12.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.198+0000 7f6f1bfff640 1 -- 172.21.15.47:0/112831919 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6f24004a90 con 0x7f6f340ac5a0 2024-03-29T11:21:12.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.198+0000 7f6f1bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:12.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.198+0000 7f6f1bfff640 1 -- 172.21.15.47:0/112831919 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f6f24001250 con 0x7f6f340ac5a0 2024-03-29T11:21:12.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.198+0000 7f6f1bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:12.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.198+0000 7f6f1bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:12.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.198+0000 7f6f1bfff640 10 monclient: dump: 2024-03-29T11:21:12.177 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:21:12.177 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:21:12.177 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:12.177 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:12.177 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:12.177 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:21:12.177 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:21:12.177 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:21:12.177 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:21:12.177 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:21:12.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.198+0000 7f6f31ffb640 4 set_mon_vals no callback set 2024-03-29T11:21:12.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.199+0000 7f6f39e0d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:12.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.200+0000 7f6f39e0d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:12.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.200+0000 7f6f39e0d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:21:12.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.200+0000 7f6f39e0d640 1 -- 172.21.15.47:0/112831919 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f14005080 con 0x7f6f340ac5a0 2024-03-29T11:21:12.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.200+0000 7f6f1bfff640 1 -- 172.21.15.47:0/112831919 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7f6f24004ee0 con 0x7f6f340ac5a0 2024-03-29T11:21:12.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.201+0000 7f6f1bfff640 1 -- 172.21.15.47:0/112831919 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7f6f24084c20 con 0x7f6f340ac5a0 2024-03-29T11:21:12.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.201+0000 7f6f1bfff640 10 client.4565.objecter ms_dispatch 0x7f6f34000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:12.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.201+0000 7f6f1bfff640 3 client.4565.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:12.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.201+0000 7f6f1bfff640 3 client.4565.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:12.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.201+0000 7f6f1bfff640 20 client.4565.objecter dump_active .. 0 homeless 2024-03-29T11:21:12.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.202+0000 7f6f32ffd640 10 monclient: get_auth_request con 0x7f6f2006db10 auth_method 0 2024-03-29T11:21:12.180 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.204+0000 7f6f1bfff640 10 client.4565.objecter ms_handle_connect 0x7f6f2006db10 2024-03-29T11:21:12.189 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.213+0000 7f6f1bfff640 1 -- 172.21.15.47:0/112831919 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6f24050560 con 0x7f6f340ac5a0 2024-03-29T11:21:12.189 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.213+0000 7f6f1bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:12.189 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.213+0000 7f6f1bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:12.537 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.560+0000 7f6f39e0d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:12.537 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.560+0000 7f6f39e0d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:12.537 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.561+0000 7f6f39e0d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:21:12.537 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.561+0000 7f6f39e0d640 1 -- 172.21.15.47:0/112831919 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6f14005330 con 0x7f6f340ac5a0 2024-03-29T11:21:12.539 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.562+0000 7f6f1bfff640 1 -- 172.21.15.47:0/112831919 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7f6f24052490 con 0x7f6f340ac5a0 2024-03-29T11:21:12.539 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.563+0000 7f6f1bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:12.539 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.563+0000 7f6f1bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:12.540 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:21:12.540 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:12.544 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.567+0000 7f6f39e0d640 10 librados: watch_flush enter 2024-03-29T11:21:12.544 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.567+0000 7f6f39e0d640 10 librados: watch_flush exit 2024-03-29T11:21:12.544 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.567+0000 7f6f39e0d640 20 client.4565.objecter shutdown clearing up homeless session... 2024-03-29T11:21:12.544 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.567+0000 7f6f39e0d640 10 client.4565.objecter successfully canceled tick 2024-03-29T11:21:12.544 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.567+0000 7f6f39e0d640 1 -- 172.21.15.47:0/112831919 >> v1:172.21.15.121:6816/1964117370 conn(0x7f6f2006db10 legacy=0x7f6f2006ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:12.544 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.567+0000 7f6f39e0d640 10 monclient: shutdown 2024-03-29T11:21:12.544 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.567+0000 7f6f39e0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:12.544 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.567+0000 7f6f39e0d640 1 -- 172.21.15.47:0/112831919 >> v1:172.21.15.47:6790/0 conn(0x7f6f340ac5a0 legacy=0x7f6f34126db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:12.546 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.568+0000 7f6f39e0d640 1 -- 172.21.15.47:0/112831919 shutdown_connections 2024-03-29T11:21:12.546 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.568+0000 7f6f39e0d640 1 -- 172.21.15.47:0/112831919 >> 172.21.15.47:0/112831919 conn(0x7f6f3406b660 msgr2=0x7f6f340b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:12.546 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.569+0000 7f6f39e0d640 1 -- 172.21.15.47:0/112831919 shutdown_connections 2024-03-29T11:21:12.546 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.569+0000 7f6f39e0d640 1 -- 172.21.15.47:0/112831919 wait complete. 2024-03-29T11:21:12.546 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:12.569+0000 7f6f39e0d640 1 librados: shutdown 2024-03-29T11:21:12.563 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:19.369 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:19.527 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.550+0000 7f9543294640 10 monclient: get_monmap_and_config 2024-03-29T11:21:19.527 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.550+0000 7f9543294640 10 monclient: build_initial_monmap 2024-03-29T11:21:19.527 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.550+0000 7f9543294640 10 monclient: monmap: 2024-03-29T11:21:19.527 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:21:19.527 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:21:19.527 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:21:19.551690+0000 2024-03-29T11:21:19.527 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:21:19.551690+0000 2024-03-29T11:21:19.527 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:19.528 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:21:19.528 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:21:19.528 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:21:19.528 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:21:19.528 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:21:19.531 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.555+0000 7f9543294640 1 Processor -- start 2024-03-29T11:21:19.532 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.555+0000 7f9543294640 1 -- start start 2024-03-29T11:21:19.532 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.555+0000 7f9543294640 10 monclient: init 2024-03-29T11:21:19.533 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.556+0000 7f9543294640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:19.533 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.556+0000 7f9543294640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:21:19.534 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.556+0000 7f9543294640 10 monclient(hunting): picked mon.noname-c con 0x7f95340389e0 addr v1:172.21.15.121:6789/0 2024-03-29T11:21:19.534 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.556+0000 7f9543294640 10 monclient(hunting): picked mon.noname-b con 0x7f9534038dc0 addr v1:172.21.15.47:6790/0 2024-03-29T11:21:19.534 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.557+0000 7f9543294640 10 monclient(hunting): picked mon.noname-a con 0x7f9534030660 addr v1:172.21.15.47:6789/0 2024-03-29T11:21:19.534 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.557+0000 7f9543294640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95340c54c0 con 0x7f9534030660 2024-03-29T11:21:19.534 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.557+0000 7f9543294640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9534036430 con 0x7f95340389e0 2024-03-29T11:21:19.534 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.557+0000 7f9543294640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9534039840 con 0x7f9534038dc0 2024-03-29T11:21:19.534 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.557+0000 7f9543294640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:19.534 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.557+0000 7f9543294640 10 monclient(hunting): authenticate will time out at 1322.745435s 2024-03-29T11:21:19.536 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.559+0000 7f9542a93640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f9534030660 0x7f95340d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:59872/0 (socket says 172.21.15.47:59872) 2024-03-29T11:21:19.536 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.559+0000 7f9542a93640 1 -- 172.21.15.47:0/2397830229 learned_addr learned my addr 172.21.15.47:0/2397830229 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:21:19.537 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.560+0000 7f9541290640 1 -- 172.21.15.47:0/2397830229 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1845230197 0 0) 0x7f95340c54c0 con 0x7f9534030660 2024-03-29T11:21:19.537 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.560+0000 7f9541290640 10 monclient(hunting): _negotiate 2024-03-29T11:21:19.537 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.560+0000 7f9541290640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:19.537 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.560+0000 7f9541290640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:19.537 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.560+0000 7f9541290640 10 monclient(hunting): my global_id is 4506 2024-03-29T11:21:19.537 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.561+0000 7f9541290640 1 -- 172.21.15.47:0/2397830229 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9528003520 con 0x7f9534030660 2024-03-29T11:21:19.538 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.561+0000 7f9541290640 1 -- 172.21.15.47:0/2397830229 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3607882482 0 0) 0x7f9528003520 con 0x7f9534030660 2024-03-29T11:21:19.538 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.561+0000 7f9541290640 1 -- 172.21.15.47:0/2397830229 >> v1:172.21.15.47:6790/0 conn(0x7f9534038dc0 legacy=0x7f95340cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:19.538 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.561+0000 7f9541290640 1 -- 172.21.15.47:0/2397830229 >> v1:172.21.15.121:6789/0 conn(0x7f95340389e0 legacy=0x7f9534033520 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:21:19.539 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.562+0000 7f9541290640 10 monclient: _finish_hunting 0 2024-03-29T11:21:19.539 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.562+0000 7f9541290640 1 monclient: found mon.noname-a 2024-03-29T11:21:19.539 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.562+0000 7f9541290640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:19.539 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.562+0000 7f9541290640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-29T11:21:19.539 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.562+0000 7f9541290640 1 -- 172.21.15.47:0/2397830229 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9534035690 con 0x7f9534030660 2024-03-29T11:21:19.539 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.562+0000 7f9541290640 10 monclient: _finish_auth 0 2024-03-29T11:21:19.539 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.562+0000 7f9541290640 10 monclient: _check_auth_tickets 2024-03-29T11:21:19.539 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.562+0000 7f9541290640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:19.539 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.562+0000 7f9541290640 1 -- 172.21.15.47:0/2397830229 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f953c04f730 con 0x7f9534030660 2024-03-29T11:21:19.540 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.562+0000 7f9541290640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:19.540 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.562+0000 7f9541290640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:19.540 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.562+0000 7f9541290640 10 monclient: dump: 2024-03-29T11:21:19.540 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:21:19.540 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:21:19.540 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:19.540 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:19.540 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:19.540 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:21:19.540 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:21:19.540 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:21:19.540 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:21:19.540 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:21:19.541 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.564+0000 7f9543294640 5 monclient: authenticate success, global_id 4506 2024-03-29T11:21:19.542 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.564+0000 7f9543294640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:19.542 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.564+0000 7f9541290640 1 -- 172.21.15.47:0/2397830229 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f953c0008d0 con 0x7f9534030660 2024-03-29T11:21:19.542 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.564+0000 7f9541290640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:19.542 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.564+0000 7f9541290640 1 -- 172.21.15.47:0/2397830229 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f953c0758b0 con 0x7f9534030660 2024-03-29T11:21:19.542 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.564+0000 7f9541290640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:19.542 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.565+0000 7f9541290640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:19.542 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.565+0000 7f9541290640 10 monclient: dump: 2024-03-29T11:21:19.542 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:21:19.542 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:21:19.542 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:19.542 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:19.542 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:19.543 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:21:19.543 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:21:19.543 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:21:19.543 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:21:19.543 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:21:19.543 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.565+0000 7f9543294640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:19.543 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.565+0000 7f9543294640 4 set_mon_vals no callback set 2024-03-29T11:21:19.543 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.566+0000 7f9543294640 10 monclient: shutdown 2024-03-29T11:21:19.544 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.566+0000 7f9543294640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:19.544 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.566+0000 7f9543294640 1 -- 172.21.15.47:0/2397830229 >> v1:172.21.15.47:6789/0 conn(0x7f9534030660 legacy=0x7f95340d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:19.545 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.569+0000 7f9543294640 1 -- 172.21.15.47:0/2397830229 shutdown_connections 2024-03-29T11:21:19.545 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.569+0000 7f9543294640 1 -- 172.21.15.47:0/2397830229 >> 172.21.15.47:0/2397830229 conn(0x7f9534005c00 msgr2=0x7f9534006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:19.546 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.569+0000 7f9543294640 1 -- 172.21.15.47:0/2397830229 shutdown_connections 2024-03-29T11:21:19.546 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.569+0000 7f9543294640 1 -- 172.21.15.47:0/2397830229 wait complete. 2024-03-29T11:21:19.547 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.570+0000 7f9543294640 10 monclient: build_initial_monmap 2024-03-29T11:21:19.547 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.570+0000 7f9543294640 10 monclient: monmap: 2024-03-29T11:21:19.547 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:21:19.547 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:19.547 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:21:19.547 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:21:19.547 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:19.547 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:21:19.547 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:21:19.547 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:21:19.547 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:21:19.547 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:21:19.549 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.571+0000 7f9543294640 1 librados: starting msgr at 2024-03-29T11:21:19.549 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.571+0000 7f9543294640 1 librados: starting objecter 2024-03-29T11:21:19.549 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.571+0000 7f9543294640 1 Processor -- start 2024-03-29T11:21:19.549 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.571+0000 7f9543294640 1 -- start start 2024-03-29T11:21:19.549 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.571+0000 7f9543294640 1 librados: setting wanted keys 2024-03-29T11:21:19.549 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.571+0000 7f9543294640 1 librados: calling monclient init 2024-03-29T11:21:19.549 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.571+0000 7f9543294640 10 monclient: init 2024-03-29T11:21:19.549 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.572+0000 7f9543294640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:19.549 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.572+0000 7f9543294640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:21:19.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.574+0000 7f9543294640 10 monclient(hunting): picked mon.noname-c con 0x7f9534030660 addr v1:172.21.15.47:6790/0 2024-03-29T11:21:19.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.574+0000 7f9543294640 10 monclient(hunting): picked mon.noname-b con 0x7f95340389e0 addr v1:172.21.15.121:6789/0 2024-03-29T11:21:19.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.574+0000 7f9543294640 10 monclient(hunting): picked mon.noname-a con 0x7f95340c5b10 addr v1:172.21.15.47:6789/0 2024-03-29T11:21:19.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.574+0000 7f9543294640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95340c73a0 con 0x7f95340c5b10 2024-03-29T11:21:19.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.574+0000 7f9543294640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95340c6080 con 0x7f95340389e0 2024-03-29T11:21:19.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.574+0000 7f9543294640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f953415a590 con 0x7f9534030660 2024-03-29T11:21:19.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.574+0000 7f9543294640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:19.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.574+0000 7f9543294640 10 monclient(hunting): authenticate will time out at 1322.762669s 2024-03-29T11:21:19.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.575+0000 7f9542a93640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f95340c5b10 0x7f953415d2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:34130/0 (socket says 172.21.15.47:34130) 2024-03-29T11:21:19.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.575+0000 7f9542a93640 1 -- 172.21.15.47:0/994925000 learned_addr learned my addr 172.21.15.47:0/994925000 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:21:19.556 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.579+0000 7f952effd640 10 client.?.objecter ms_handle_connect 0x7f95340c5b10 2024-03-29T11:21:19.556 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.579+0000 7f952effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:19.556 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.580+0000 7f952effd640 10 client.?.objecter ms_handle_connect 0x7f95340389e0 2024-03-29T11:21:19.563 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.580+0000 7f952effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:19.563 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.580+0000 7f952effd640 1 -- 172.21.15.47:0/994925000 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1221018922 0 0) 0x7f95340c73a0 con 0x7f95340c5b10 2024-03-29T11:21:19.563 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.580+0000 7f952effd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:19.563 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.580+0000 7f952effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:19.563 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.580+0000 7f952effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:19.563 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.580+0000 7f952effd640 10 monclient(hunting): my global_id is 4509 2024-03-29T11:21:19.563 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.580+0000 7f952effd640 1 -- 172.21.15.47:0/994925000 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9520004eb0 con 0x7f95340c5b10 2024-03-29T11:21:19.563 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.581+0000 7f952effd640 1 -- 172.21.15.47:0/994925000 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2579752305 0 0) 0x7f95340c6080 con 0x7f95340389e0 2024-03-29T11:21:19.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.581+0000 7f952effd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:19.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.581+0000 7f952effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:19.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.581+0000 7f952effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:19.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.581+0000 7f952effd640 10 monclient(hunting): my global_id is 4495 2024-03-29T11:21:19.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.581+0000 7f952effd640 1 -- 172.21.15.47:0/994925000 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95340c73a0 con 0x7f95340389e0 2024-03-29T11:21:19.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.581+0000 7f952effd640 1 -- 172.21.15.47:0/994925000 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2142845245 0 0) 0x7f9520004eb0 con 0x7f95340c5b10 2024-03-29T11:21:19.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.581+0000 7f952effd640 1 -- 172.21.15.47:0/994925000 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95340c6080 con 0x7f95340c5b10 2024-03-29T11:21:19.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.581+0000 7f952effd640 1 -- 172.21.15.47:0/994925000 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f953c04f400 con 0x7f95340c5b10 2024-03-29T11:21:19.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.581+0000 7f952effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:19.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.581+0000 7f952effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:19.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.581+0000 7f952effd640 10 monclient(hunting): dump: 2024-03-29T11:21:19.564 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:21:19.564 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:21:19.564 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:19.565 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:19.565 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:19.565 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:21:19.565 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:21:19.565 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:21:19.565 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:21:19.565 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:21:19.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.581+0000 7f952effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:19.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.581+0000 7f952effd640 10 client.?.objecter ms_handle_connect 0x7f9534030660 2024-03-29T11:21:19.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.581+0000 7f952effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:19.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.581+0000 7f952effd640 1 -- 172.21.15.47:0/994925000 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3703860821 0 0) 0x7f95340c73a0 con 0x7f95340389e0 2024-03-29T11:21:19.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.581+0000 7f952effd640 1 -- 172.21.15.47:0/994925000 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9520004eb0 con 0x7f95340389e0 2024-03-29T11:21:19.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.581+0000 7f952effd640 1 -- 172.21.15.47:0/994925000 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f9530001890 con 0x7f95340389e0 2024-03-29T11:21:19.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.581+0000 7f952effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:19.566 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.581+0000 7f952effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:19.566 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.581+0000 7f952effd640 10 monclient(hunting): dump: 2024-03-29T11:21:19.566 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:21:19.566 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:21:19.566 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:19.566 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:19.566 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:19.566 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:21:19.566 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:21:19.566 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:21:19.566 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:21:19.566 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:21:19.566 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.582+0000 7f952effd640 1 -- 172.21.15.47:0/994925000 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2543965611 0 0) 0x7f95340c6080 con 0x7f95340c5b10 2024-03-29T11:21:19.566 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.582+0000 7f952effd640 1 -- 172.21.15.47:0/994925000 >> v1:172.21.15.47:6790/0 conn(0x7f9534030660 legacy=0x7f953402fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:19.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.582+0000 7f952effd640 1 -- 172.21.15.47:0/994925000 >> v1:172.21.15.121:6789/0 conn(0x7f95340389e0 legacy=0x7f953415cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:19.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.582+0000 7f952effd640 10 monclient: _finish_hunting 0 2024-03-29T11:21:19.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.582+0000 7f952effd640 1 monclient: found mon.a 2024-03-29T11:21:19.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.582+0000 7f952effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:19.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.582+0000 7f952effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:21:19.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.582+0000 7f952effd640 1 -- 172.21.15.47:0/994925000 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f953416b620 con 0x7f95340c5b10 2024-03-29T11:21:19.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.582+0000 7f952effd640 10 monclient: _finish_auth 0 2024-03-29T11:21:19.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.582+0000 7f952effd640 10 monclient: _check_auth_tickets 2024-03-29T11:21:19.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.582+0000 7f952effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:19.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.582+0000 7f9543294640 5 monclient: authenticate success, global_id 4509 2024-03-29T11:21:19.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.582+0000 7f9543294640 10 monclient: _renew_subs 2024-03-29T11:21:19.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.582+0000 7f9543294640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:21:19.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.582+0000 7f9543294640 1 -- 172.21.15.47:0/994925000 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95341685f0 con 0x7f95340c5b10 2024-03-29T11:21:19.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.582+0000 7f9543294640 10 client.4509.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:19.567 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.582+0000 7f9543294640 10 monclient: _renew_subs 2024-03-29T11:21:19.568 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.582+0000 7f9543294640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:21:19.568 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.582+0000 7f9543294640 1 -- 172.21.15.47:0/994925000 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9534168a80 con 0x7f95340c5b10 2024-03-29T11:21:19.568 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.582+0000 7f9543294640 1 librados: init done 2024-03-29T11:21:19.568 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.582+0000 7f952effd640 1 -- 172.21.15.47:0/994925000 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f953c04fdd0 con 0x7f95340c5b10 2024-03-29T11:21:19.568 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.583+0000 7f952effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:19.568 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.583+0000 7f952effd640 1 -- 172.21.15.47:0/994925000 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f953c067bf0 con 0x7f95340c5b10 2024-03-29T11:21:19.568 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.583+0000 7f952effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:19.568 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.583+0000 7f952effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:19.568 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.583+0000 7f952effd640 10 monclient: dump: 2024-03-29T11:21:19.568 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:21:19.568 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:21:19.568 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:19.568 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:19.568 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:19.569 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:21:19.569 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:21:19.569 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:21:19.569 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:21:19.569 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:21:19.569 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.583+0000 7f9540a8f640 4 set_mon_vals no callback set 2024-03-29T11:21:19.569 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.585+0000 7f952effd640 1 -- 172.21.15.47:0/994925000 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7f953c067d50 con 0x7f95340c5b10 2024-03-29T11:21:19.569 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.585+0000 7f952effd640 1 -- 172.21.15.47:0/994925000 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7f953c0ff530 con 0x7f95340c5b10 2024-03-29T11:21:19.569 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.585+0000 7f952effd640 10 client.4509.objecter ms_dispatch 0x7f953c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:19.569 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.585+0000 7f952effd640 3 client.4509.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:19.569 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.585+0000 7f952effd640 3 client.4509.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:19.569 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.585+0000 7f952effd640 20 client.4509.objecter dump_active .. 0 homeless 2024-03-29T11:21:19.569 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.586+0000 7f9542292640 10 monclient: get_auth_request con 0x7f952006dd30 auth_method 0 2024-03-29T11:21:19.569 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.586+0000 7f9543294640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:19.569 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.586+0000 7f9543294640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:19.570 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.586+0000 7f9543294640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:21:19.570 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.586+0000 7f9543294640 1 -- 172.21.15.47:0/994925000 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9524005080 con 0x7f95340c5b10 2024-03-29T11:21:19.570 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.587+0000 7f952effd640 10 client.4509.objecter ms_handle_connect 0x7f952006dd30 2024-03-29T11:21:19.573 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.597+0000 7f952effd640 1 -- 172.21.15.47:0/994925000 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f953c0cae10 con 0x7f95340c5b10 2024-03-29T11:21:19.573 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.597+0000 7f952effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:19.573 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.597+0000 7f952effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:19.895 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.919+0000 7f9543294640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:19.895 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.919+0000 7f9543294640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:19.896 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.919+0000 7f9543294640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:21:19.896 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.919+0000 7f9543294640 1 -- 172.21.15.47:0/994925000 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9524005330 con 0x7f95340c5b10 2024-03-29T11:21:19.897 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.920+0000 7f952effd640 1 -- 172.21.15.47:0/994925000 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7f953c0ccd40 con 0x7f95340c5b10 2024-03-29T11:21:19.897 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.920+0000 7f952effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:19.897 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.920+0000 7f952effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:19.900 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:21:19.900 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:19.903 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.926+0000 7f9543294640 10 librados: watch_flush enter 2024-03-29T11:21:19.903 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.926+0000 7f9543294640 10 librados: watch_flush exit 2024-03-29T11:21:19.903 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.926+0000 7f9543294640 20 client.4509.objecter shutdown clearing up homeless session... 2024-03-29T11:21:19.903 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.926+0000 7f9543294640 10 client.4509.objecter successfully canceled tick 2024-03-29T11:21:19.903 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.926+0000 7f9543294640 1 -- 172.21.15.47:0/994925000 >> v1:172.21.15.121:6816/1964117370 conn(0x7f952006dd30 legacy=0x7f95200701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:19.903 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.926+0000 7f9543294640 10 monclient: shutdown 2024-03-29T11:21:19.903 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.926+0000 7f9543294640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:19.903 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.926+0000 7f9543294640 1 -- 172.21.15.47:0/994925000 >> v1:172.21.15.47:6789/0 conn(0x7f95340c5b10 legacy=0x7f953415d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:19.903 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.926+0000 7f9543294640 1 -- 172.21.15.47:0/994925000 shutdown_connections 2024-03-29T11:21:19.904 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.926+0000 7f9543294640 1 -- 172.21.15.47:0/994925000 >> 172.21.15.47:0/994925000 conn(0x7f9534005c00 msgr2=0x7f9534034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:19.904 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.927+0000 7f9543294640 1 -- 172.21.15.47:0/994925000 shutdown_connections 2024-03-29T11:21:19.904 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.927+0000 7f9543294640 1 -- 172.21.15.47:0/994925000 wait complete. 2024-03-29T11:21:19.904 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:19.928+0000 7f9543294640 1 librados: shutdown 2024-03-29T11:21:19.924 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:26.732 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:26.886 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.909+0000 7fadb457b640 10 monclient: get_monmap_and_config 2024-03-29T11:21:26.886 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.909+0000 7fadb457b640 10 monclient: build_initial_monmap 2024-03-29T11:21:26.886 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.909+0000 7fadb457b640 10 monclient: monmap: 2024-03-29T11:21:26.886 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:21:26.886 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:21:26.886 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:21:26.910608+0000 2024-03-29T11:21:26.886 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:21:26.910608+0000 2024-03-29T11:21:26.886 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:26.886 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:21:26.886 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:21:26.886 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:21:26.887 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:21:26.887 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:21:26.887 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.910+0000 7fadb457b640 1 Processor -- start 2024-03-29T11:21:26.887 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.910+0000 7fadb457b640 1 -- start start 2024-03-29T11:21:26.887 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.910+0000 7fadb457b640 10 monclient: init 2024-03-29T11:21:26.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.911+0000 7fadb457b640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:26.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.911+0000 7fadb457b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:21:26.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.911+0000 7fadb457b640 10 monclient(hunting): picked mon.noname-a con 0x7fadac1422b0 addr v1:172.21.15.47:6789/0 2024-03-29T11:21:26.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.911+0000 7fadb457b640 10 monclient(hunting): picked mon.noname-b con 0x7fadac142690 addr v1:172.21.15.47:6790/0 2024-03-29T11:21:26.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.911+0000 7fadb457b640 10 monclient(hunting): picked mon.noname-c con 0x7fadac0b17d0 addr v1:172.21.15.121:6789/0 2024-03-29T11:21:26.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.911+0000 7fadb457b640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadac142c80 con 0x7fadac1422b0 2024-03-29T11:21:26.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.911+0000 7fadb457b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadac0aae40 con 0x7fadac0b17d0 2024-03-29T11:21:26.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.911+0000 7fadb457b640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadac13fbd0 con 0x7fadac142690 2024-03-29T11:21:26.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.911+0000 7fadb457b640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:26.889 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.911+0000 7fadb457b640 10 monclient(hunting): authenticate will time out at 1330.099650s 2024-03-29T11:21:26.889 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.911+0000 7fadb2317640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fadac1422b0 0x7fadac13f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:51788/0 (socket says 172.21.15.47:51788) 2024-03-29T11:21:26.889 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.911+0000 7fadb2317640 1 -- 172.21.15.47:0/67109522 learned_addr learned my addr 172.21.15.47:0/67109522 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:21:26.890 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.912+0000 7fadb1315640 1 -- 172.21.15.47:0/67109522 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3483019934 0 0) 0x7fadac142c80 con 0x7fadac1422b0 2024-03-29T11:21:26.890 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.912+0000 7fadb1315640 10 monclient(hunting): _negotiate 2024-03-29T11:21:26.890 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.912+0000 7fadb1315640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:26.890 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.912+0000 7fadb1315640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:26.890 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.912+0000 7fadb1315640 10 monclient(hunting): my global_id is 4512 2024-03-29T11:21:26.890 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.912+0000 7fadb1315640 1 -- 172.21.15.47:0/67109522 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad98003520 con 0x7fadac1422b0 2024-03-29T11:21:26.891 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.912+0000 7fadb1315640 1 -- 172.21.15.47:0/67109522 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1410534484 0 0) 0x7fadac0aae40 con 0x7fadac0b17d0 2024-03-29T11:21:26.891 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.912+0000 7fadb1315640 10 monclient(hunting): _negotiate 2024-03-29T11:21:26.891 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.912+0000 7fadb1315640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:26.891 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.912+0000 7fadb1315640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:26.891 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.912+0000 7fadb1315640 10 monclient(hunting): my global_id is 4498 2024-03-29T11:21:26.891 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.912+0000 7fadb1315640 1 -- 172.21.15.47:0/67109522 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadac142c80 con 0x7fadac0b17d0 2024-03-29T11:21:26.891 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.912+0000 7fadb1315640 1 -- 172.21.15.47:0/67109522 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3020256535 0 0) 0x7fad98003520 con 0x7fadac1422b0 2024-03-29T11:21:26.891 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.912+0000 7fadb1315640 1 -- 172.21.15.47:0/67109522 >> v1:172.21.15.47:6790/0 conn(0x7fadac142690 legacy=0x7fadac0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:26.891 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.912+0000 7fadb1315640 1 -- 172.21.15.47:0/67109522 >> v1:172.21.15.121:6789/0 conn(0x7fadac0b17d0 legacy=0x7fadac0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:26.891 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.912+0000 7fadb1315640 10 monclient: _finish_hunting 0 2024-03-29T11:21:26.891 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.912+0000 7fadb1315640 1 monclient: found mon.noname-a 2024-03-29T11:21:26.891 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.912+0000 7fadb1315640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:26.891 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.912+0000 7fadb1315640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-29T11:21:26.891 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.912+0000 7fadb1315640 1 -- 172.21.15.47:0/67109522 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadac151940 con 0x7fadac1422b0 2024-03-29T11:21:26.891 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.912+0000 7fadb1315640 10 monclient: _finish_auth 0 2024-03-29T11:21:26.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.912+0000 7fadb1315640 10 monclient: _check_auth_tickets 2024-03-29T11:21:26.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.912+0000 7fadb1315640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:26.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.912+0000 7fadb1315640 1 -- 172.21.15.47:0/67109522 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fada8001610 con 0x7fadac1422b0 2024-03-29T11:21:26.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.912+0000 7fadb457b640 5 monclient: authenticate success, global_id 4512 2024-03-29T11:21:26.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.912+0000 7fadb457b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:26.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.912+0000 7fadb1315640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:26.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.912+0000 7fadb1315640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:26.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.912+0000 7fadb1315640 10 monclient: dump: 2024-03-29T11:21:26.892 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:21:26.892 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:21:26.892 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:26.892 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:26.892 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:26.892 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:21:26.893 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:21:26.893 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:21:26.893 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:21:26.893 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:21:26.893 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.912+0000 7fadb457b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:26.893 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.913+0000 7fadb1315640 1 -- 172.21.15.47:0/67109522 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fada8001770 con 0x7fadac1422b0 2024-03-29T11:21:26.893 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.913+0000 7fadb1315640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:26.893 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.913+0000 7fadb1315640 1 -- 172.21.15.47:0/67109522 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fada8001b80 con 0x7fadac1422b0 2024-03-29T11:21:26.893 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.913+0000 7fadb1315640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:26.893 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.913+0000 7fadb1315640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:26.893 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.913+0000 7fadb1315640 10 monclient: dump: 2024-03-29T11:21:26.893 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:21:26.893 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:21:26.893 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:26.893 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:26.894 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:26.894 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:21:26.894 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:21:26.894 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:21:26.894 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:21:26.894 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:21:26.894 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.913+0000 7fadb457b640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:26.894 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.913+0000 7fadb457b640 4 set_mon_vals no callback set 2024-03-29T11:21:26.894 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.913+0000 7fadb457b640 10 monclient: shutdown 2024-03-29T11:21:26.894 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.913+0000 7fadb457b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:26.894 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.913+0000 7fadb457b640 1 -- 172.21.15.47:0/67109522 >> v1:172.21.15.47:6789/0 conn(0x7fadac1422b0 legacy=0x7fadac13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:26.894 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.914+0000 7fadb457b640 1 -- 172.21.15.47:0/67109522 shutdown_connections 2024-03-29T11:21:26.894 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.914+0000 7fadb457b640 1 -- 172.21.15.47:0/67109522 >> 172.21.15.47:0/67109522 conn(0x7fadac066a00 msgr2=0x7fadac066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:26.898 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.914+0000 7fadb457b640 1 -- 172.21.15.47:0/67109522 shutdown_connections 2024-03-29T11:21:26.898 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.914+0000 7fadb457b640 1 -- 172.21.15.47:0/67109522 wait complete. 2024-03-29T11:21:26.898 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.915+0000 7fadb457b640 10 monclient: build_initial_monmap 2024-03-29T11:21:26.899 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.915+0000 7fadb457b640 10 monclient: monmap: 2024-03-29T11:21:26.899 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:21:26.899 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:26.899 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:21:26.899 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:21:26.899 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:26.899 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:21:26.899 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:21:26.899 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:21:26.899 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:21:26.899 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:21:26.899 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.915+0000 7fadb457b640 1 librados: starting msgr at 2024-03-29T11:21:26.899 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.915+0000 7fadb457b640 1 librados: starting objecter 2024-03-29T11:21:26.899 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.915+0000 7fadb457b640 1 Processor -- start 2024-03-29T11:21:26.900 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.915+0000 7fadb457b640 1 -- start start 2024-03-29T11:21:26.900 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.915+0000 7fadb457b640 1 librados: setting wanted keys 2024-03-29T11:21:26.900 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.915+0000 7fadb457b640 1 librados: calling monclient init 2024-03-29T11:21:26.900 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.915+0000 7fadb457b640 10 monclient: init 2024-03-29T11:21:26.900 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.916+0000 7fadb457b640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:26.900 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.916+0000 7fadb457b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:21:26.900 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.916+0000 7fadb457b640 10 monclient(hunting): picked mon.noname-c con 0x7fadac0b17d0 addr v1:172.21.15.47:6790/0 2024-03-29T11:21:26.900 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.916+0000 7fadb457b640 10 monclient(hunting): picked mon.noname-b con 0x7fadac1422b0 addr v1:172.21.15.121:6789/0 2024-03-29T11:21:26.900 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.916+0000 7fadb457b640 10 monclient(hunting): picked mon.noname-a con 0x7fadac142690 addr v1:172.21.15.47:6789/0 2024-03-29T11:21:26.900 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.916+0000 7fadb457b640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadac072090 con 0x7fadac142690 2024-03-29T11:21:26.900 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.916+0000 7fadb457b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadac06f870 con 0x7fadac1422b0 2024-03-29T11:21:26.900 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.916+0000 7fadb457b640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadac06f230 con 0x7fadac0b17d0 2024-03-29T11:21:26.900 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.916+0000 7fadb457b640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:26.900 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.916+0000 7fadb457b640 10 monclient(hunting): authenticate will time out at 1330.104481s 2024-03-29T11:21:26.901 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.916+0000 7fadb2317640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fadac0b17d0 0x7fadac071980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:39218/0 (socket says 172.21.15.47:39218) 2024-03-29T11:21:26.901 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.916+0000 7fadb2317640 1 -- 172.21.15.47:0/2657638470 learned_addr learned my addr 172.21.15.47:0/2657638470 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:21:26.901 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.916+0000 7fada2ffd640 10 client.?.objecter ms_handle_connect 0x7fadac0b17d0 2024-03-29T11:21:26.901 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.916+0000 7fada2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:26.901 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.916+0000 7fada2ffd640 10 client.?.objecter ms_handle_connect 0x7fadac1422b0 2024-03-29T11:21:26.901 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.916+0000 7fada2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:26.901 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.916+0000 7fada2ffd640 1 -- 172.21.15.47:0/2657638470 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 190628730 0 0) 0x7fadac06f230 con 0x7fadac0b17d0 2024-03-29T11:21:26.901 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.917+0000 7fada2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:26.901 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.917+0000 7fada2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:26.901 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.917+0000 7fada2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:26.901 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.917+0000 7fada2ffd640 10 monclient(hunting): my global_id is 4574 2024-03-29T11:21:26.901 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.917+0000 7fada2ffd640 1 -- 172.21.15.47:0/2657638470 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad90004eb0 con 0x7fadac0b17d0 2024-03-29T11:21:26.901 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.917+0000 7fada2ffd640 10 client.?.objecter ms_handle_connect 0x7fadac142690 2024-03-29T11:21:26.901 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.917+0000 7fada2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:26.901 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.917+0000 7fada2ffd640 1 -- 172.21.15.47:0/2657638470 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1988674922 0 0) 0x7fadac06f870 con 0x7fadac1422b0 2024-03-29T11:21:26.902 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.917+0000 7fada2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:26.902 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.917+0000 7fada2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:26.902 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.917+0000 7fada2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:26.902 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.917+0000 7fada2ffd640 10 monclient(hunting): my global_id is 4501 2024-03-29T11:21:26.902 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.917+0000 7fada2ffd640 1 -- 172.21.15.47:0/2657638470 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadac06f230 con 0x7fadac1422b0 2024-03-29T11:21:26.902 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.917+0000 7fada2ffd640 1 -- 172.21.15.47:0/2657638470 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1371816314 0 0) 0x7fad90004eb0 con 0x7fadac0b17d0 2024-03-29T11:21:26.902 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.917+0000 7fada2ffd640 1 -- 172.21.15.47:0/2657638470 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fadac06f870 con 0x7fadac0b17d0 2024-03-29T11:21:26.902 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.917+0000 7fada2ffd640 1 -- 172.21.15.47:0/2657638470 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fada8001b40 con 0x7fadac0b17d0 2024-03-29T11:21:26.902 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.917+0000 7fada2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:26.902 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.917+0000 7fada2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:26.902 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.917+0000 7fada2ffd640 10 monclient(hunting): dump: 2024-03-29T11:21:26.902 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:21:26.902 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:21:26.902 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:26.903 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:26.903 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:26.903 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:21:26.903 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:21:26.903 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:21:26.903 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:21:26.903 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:21:26.903 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.917+0000 7fada2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:26.903 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.917+0000 7fada2ffd640 1 -- 172.21.15.47:0/2657638470 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3899387118 0 0) 0x7fadac06f230 con 0x7fadac1422b0 2024-03-29T11:21:26.903 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.917+0000 7fada2ffd640 1 -- 172.21.15.47:0/2657638470 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad90004eb0 con 0x7fadac1422b0 2024-03-29T11:21:26.903 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.917+0000 7fada2ffd640 1 -- 172.21.15.47:0/2657638470 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fad9c001800 con 0x7fadac1422b0 2024-03-29T11:21:26.903 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.917+0000 7fada2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:26.903 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.917+0000 7fada2ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:26.903 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.918+0000 7fada2ffd640 10 monclient(hunting): dump: 2024-03-29T11:21:26.903 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:21:26.904 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:21:26.904 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:26.904 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:26.904 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:26.904 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:21:26.904 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:21:26.904 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:21:26.904 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:21:26.904 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:21:26.904 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.918+0000 7fada2ffd640 1 -- 172.21.15.47:0/2657638470 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1579310244 0 0) 0x7fadac06f870 con 0x7fadac0b17d0 2024-03-29T11:21:26.904 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.918+0000 7fada2ffd640 1 -- 172.21.15.47:0/2657638470 >> v1:172.21.15.121:6789/0 conn(0x7fadac1422b0 legacy=0x7fadac06eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:26.904 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.918+0000 7fada2ffd640 1 -- 172.21.15.47:0/2657638470 >> v1:172.21.15.47:6789/0 conn(0x7fadac142690 legacy=0x7fadac1e6270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:26.904 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.918+0000 7fada2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:21:26.904 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.918+0000 7fada2ffd640 1 monclient: found mon.c 2024-03-29T11:21:26.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.918+0000 7fada2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:26.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.918+0000 7fada2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:21:26.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.918+0000 7fada2ffd640 1 -- 172.21.15.47:0/2657638470 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadac070f10 con 0x7fadac0b17d0 2024-03-29T11:21:26.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.918+0000 7fada2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:21:26.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.918+0000 7fada2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:21:26.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.918+0000 7fada2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:26.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.918+0000 7fadb457b640 5 monclient: authenticate success, global_id 4574 2024-03-29T11:21:26.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.918+0000 7fadb457b640 10 monclient: _renew_subs 2024-03-29T11:21:26.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.918+0000 7fadb457b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:21:26.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.918+0000 7fadb457b640 1 -- 172.21.15.47:0/2657638470 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadac13c9d0 con 0x7fadac0b17d0 2024-03-29T11:21:26.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.918+0000 7fadb457b640 10 client.4574.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:26.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.918+0000 7fadb457b640 10 monclient: _renew_subs 2024-03-29T11:21:26.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.918+0000 7fadb457b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:21:26.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.918+0000 7fadb457b640 1 -- 172.21.15.47:0/2657638470 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fadac06fd20 con 0x7fadac0b17d0 2024-03-29T11:21:26.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.918+0000 7fadb457b640 1 librados: init done 2024-03-29T11:21:26.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.918+0000 7fada2ffd640 1 -- 172.21.15.47:0/2657638470 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fada8004a90 con 0x7fadac0b17d0 2024-03-29T11:21:26.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.918+0000 7fada2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:26.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.918+0000 7fada2ffd640 1 -- 172.21.15.47:0/2657638470 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fada8004e20 con 0x7fadac0b17d0 2024-03-29T11:21:26.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.918+0000 7fada2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:26.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.918+0000 7fada2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:26.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.918+0000 7fada2ffd640 10 monclient: dump: 2024-03-29T11:21:26.906 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:21:26.906 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:21:26.906 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:26.906 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:26.906 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:26.906 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:21:26.907 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:21:26.907 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:21:26.907 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:21:26.907 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:21:26.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.918+0000 7fadb0b14640 4 set_mon_vals no callback set 2024-03-29T11:21:26.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.920+0000 7fadb457b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:26.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.920+0000 7fadb457b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:26.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.920+0000 7fadb457b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:21:26.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.920+0000 7fadb457b640 1 -- 172.21.15.47:0/2657638470 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad94005080 con 0x7fadac0b17d0 2024-03-29T11:21:26.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.921+0000 7fada2ffd640 1 -- 172.21.15.47:0/2657638470 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7fada801adf0 con 0x7fadac0b17d0 2024-03-29T11:21:26.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.922+0000 7fada2ffd640 1 -- 172.21.15.47:0/2657638470 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7fada8084b60 con 0x7fadac0b17d0 2024-03-29T11:21:26.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.922+0000 7fada2ffd640 10 client.4574.objecter ms_dispatch 0x7fadac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:26.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.922+0000 7fada2ffd640 3 client.4574.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:26.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.922+0000 7fada2ffd640 3 client.4574.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:26.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.922+0000 7fada2ffd640 20 client.4574.objecter dump_active .. 0 homeless 2024-03-29T11:21:26.908 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.922+0000 7fadb1b16640 10 monclient: get_auth_request con 0x7fad9006dee0 auth_method 0 2024-03-29T11:21:26.908 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.922+0000 7fada2ffd640 10 client.4574.objecter ms_handle_connect 0x7fad9006dee0 2024-03-29T11:21:26.909 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.932+0000 7fada2ffd640 1 -- 172.21.15.47:0/2657638470 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fada8050440 con 0x7fadac0b17d0 2024-03-29T11:21:26.909 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.932+0000 7fada2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:26.909 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:26.932+0000 7fada2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:27.304 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:27.327+0000 7fadb457b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:27.304 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:27.327+0000 7fadb457b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:27.304 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:27.327+0000 7fadb457b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:21:27.304 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:27.327+0000 7fadb457b640 1 -- 172.21.15.47:0/2657638470 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fad94005330 con 0x7fadac0b17d0 2024-03-29T11:21:27.305 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:27.328+0000 7fada2ffd640 1 -- 172.21.15.47:0/2657638470 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7fada8052370 con 0x7fadac0b17d0 2024-03-29T11:21:27.305 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:27.328+0000 7fada2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:27.305 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:27.328+0000 7fada2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:27.305 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:21:27.305 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:27.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:27.331+0000 7fadb457b640 10 librados: watch_flush enter 2024-03-29T11:21:27.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:27.331+0000 7fadb457b640 10 librados: watch_flush exit 2024-03-29T11:21:27.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:27.331+0000 7fadb457b640 20 client.4574.objecter shutdown clearing up homeless session... 2024-03-29T11:21:27.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:27.331+0000 7fadb457b640 10 client.4574.objecter successfully canceled tick 2024-03-29T11:21:27.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:27.331+0000 7fadb457b640 1 -- 172.21.15.47:0/2657638470 >> v1:172.21.15.121:6816/1964117370 conn(0x7fad9006dee0 legacy=0x7fad900703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:27.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:27.331+0000 7fadb457b640 10 monclient: shutdown 2024-03-29T11:21:27.309 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:27.331+0000 7fadb457b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:27.309 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:27.331+0000 7fadb457b640 1 -- 172.21.15.47:0/2657638470 >> v1:172.21.15.47:6790/0 conn(0x7fadac0b17d0 legacy=0x7fadac071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:27.309 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:27.331+0000 7fadb457b640 1 -- 172.21.15.47:0/2657638470 shutdown_connections 2024-03-29T11:21:27.309 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:27.331+0000 7fadb457b640 1 -- 172.21.15.47:0/2657638470 >> 172.21.15.47:0/2657638470 conn(0x7fadac066a00 msgr2=0x7fadac0b3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:27.309 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:27.331+0000 7fadb457b640 1 -- 172.21.15.47:0/2657638470 shutdown_connections 2024-03-29T11:21:27.309 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:27.332+0000 7fadb457b640 1 -- 172.21.15.47:0/2657638470 wait complete. 2024-03-29T11:21:27.309 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:27.332+0000 7fadb457b640 1 librados: shutdown 2024-03-29T11:21:27.322 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:34.130 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:34.283 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.307+0000 7f4d2237b640 10 monclient: get_monmap_and_config 2024-03-29T11:21:34.284 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.307+0000 7f4d2237b640 10 monclient: build_initial_monmap 2024-03-29T11:21:34.284 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.307+0000 7f4d2237b640 10 monclient: monmap: 2024-03-29T11:21:34.284 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:21:34.284 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:21:34.284 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:21:34.308259+0000 2024-03-29T11:21:34.284 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:21:34.308259+0000 2024-03-29T11:21:34.284 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:34.284 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:21:34.284 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:21:34.284 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:21:34.284 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:21:34.284 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:21:34.285 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.308+0000 7f4d2237b640 1 Processor -- start 2024-03-29T11:21:34.285 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.308+0000 7f4d2237b640 1 -- start start 2024-03-29T11:21:34.285 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.308+0000 7f4d2237b640 10 monclient: init 2024-03-29T11:21:34.287 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.309+0000 7f4d2237b640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:34.287 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.309+0000 7f4d2237b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:21:34.287 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.310+0000 7f4d2237b640 10 monclient(hunting): picked mon.noname-b con 0x7f4d1c1422b0 addr v1:172.21.15.47:6790/0 2024-03-29T11:21:34.287 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.310+0000 7f4d2237b640 10 monclient(hunting): picked mon.noname-a con 0x7f4d1c142690 addr v1:172.21.15.47:6789/0 2024-03-29T11:21:34.287 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.310+0000 7f4d1bfff640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f4d1c1422b0 0x7f4d1c13f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:60338/0 (socket says 172.21.15.47:60338) 2024-03-29T11:21:34.287 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.310+0000 7f4d1bfff640 1 -- 172.21.15.47:0/3637869367 learned_addr learned my addr 172.21.15.47:0/3637869367 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:21:34.287 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.310+0000 7f4d2237b640 10 monclient(hunting): picked mon.noname-c con 0x7f4d1c0b17d0 addr v1:172.21.15.121:6789/0 2024-03-29T11:21:34.288 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.310+0000 7f4d2237b640 1 -- 172.21.15.47:0/3637869367 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d1c142c80 con 0x7f4d1c142690 2024-03-29T11:21:34.288 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.310+0000 7f4d2237b640 1 -- 172.21.15.47:0/3637869367 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d1c0aae40 con 0x7f4d1c0b17d0 2024-03-29T11:21:34.288 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.310+0000 7f4d2237b640 1 -- 172.21.15.47:0/3637869367 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d1c13fbd0 con 0x7f4d1c1422b0 2024-03-29T11:21:34.288 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.310+0000 7f4d2237b640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:34.288 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.310+0000 7f4d2237b640 10 monclient(hunting): authenticate will time out at 1337.498682s 2024-03-29T11:21:34.288 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.310+0000 7f4d1affd640 1 -- 172.21.15.47:0/3637869367 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4051383205 0 0) 0x7f4d1c0aae40 con 0x7f4d1c0b17d0 2024-03-29T11:21:34.288 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.310+0000 7f4d1affd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:34.288 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.310+0000 7f4d1affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:34.288 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.310+0000 7f4d1affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:34.288 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.310+0000 7f4d1affd640 10 monclient(hunting): my global_id is 4504 2024-03-29T11:21:34.288 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.310+0000 7f4d1affd640 1 -- 172.21.15.47:0/3637869367 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d04003520 con 0x7f4d1c0b17d0 2024-03-29T11:21:34.288 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.310+0000 7f4d1affd640 1 -- 172.21.15.47:0/3637869367 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 99777205 0 0) 0x7f4d1c13fbd0 con 0x7f4d1c1422b0 2024-03-29T11:21:34.289 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.310+0000 7f4d1affd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:34.289 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.310+0000 7f4d1affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:34.289 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.310+0000 7f4d1affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:34.289 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.310+0000 7f4d1affd640 10 monclient(hunting): my global_id is 4577 2024-03-29T11:21:34.289 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.310+0000 7f4d1affd640 1 -- 172.21.15.47:0/3637869367 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d1c0aae40 con 0x7f4d1c1422b0 2024-03-29T11:21:34.290 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.311+0000 7f4d1affd640 1 -- 172.21.15.47:0/3637869367 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1333660087 0 0) 0x7f4d04003520 con 0x7f4d1c0b17d0 2024-03-29T11:21:34.290 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.311+0000 7f4d1affd640 1 -- 172.21.15.47:0/3637869367 >> v1:172.21.15.47:6790/0 conn(0x7f4d1c1422b0 legacy=0x7f4d1c13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:34.290 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.311+0000 7f4d1affd640 1 -- 172.21.15.47:0/3637869367 >> v1:172.21.15.47:6789/0 conn(0x7f4d1c142690 legacy=0x7f4d1c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:34.290 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.311+0000 7f4d1affd640 10 monclient: _finish_hunting 0 2024-03-29T11:21:34.290 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.311+0000 7f4d1affd640 1 monclient: found mon.noname-c 2024-03-29T11:21:34.291 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.311+0000 7f4d1affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:34.291 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.311+0000 7f4d1affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:21:34.291 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.311+0000 7f4d1affd640 1 -- 172.21.15.47:0/3637869367 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d1c14f920 con 0x7f4d1c0b17d0 2024-03-29T11:21:34.291 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.311+0000 7f4d1affd640 10 monclient: _finish_auth 0 2024-03-29T11:21:34.291 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.311+0000 7f4d1affd640 10 monclient: _check_auth_tickets 2024-03-29T11:21:34.291 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.311+0000 7f4d1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:34.291 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.311+0000 7f4d1affd640 1 -- 172.21.15.47:0/3637869367 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f4d0c001690 con 0x7f4d1c0b17d0 2024-03-29T11:21:34.291 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.311+0000 7f4d1affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:34.291 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.311+0000 7f4d1affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:34.291 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.311+0000 7f4d1affd640 10 monclient: dump: 2024-03-29T11:21:34.291 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:21:34.291 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:21:34.291 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:34.291 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:34.292 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:34.292 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:21:34.292 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:21:34.292 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:21:34.292 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:21:34.292 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:21:34.292 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.311+0000 7f4d2237b640 5 monclient: authenticate success, global_id 4504 2024-03-29T11:21:34.292 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.311+0000 7f4d2237b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:34.292 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.312+0000 7f4d1affd640 1 -- 172.21.15.47:0/3637869367 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4d0c0017f0 con 0x7f4d1c0b17d0 2024-03-29T11:21:34.292 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.312+0000 7f4d1affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:34.292 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.312+0000 7f4d1affd640 1 -- 172.21.15.47:0/3637869367 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f4d0c001b80 con 0x7f4d1c0b17d0 2024-03-29T11:21:34.292 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.312+0000 7f4d2237b640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:34.292 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.312+0000 7f4d2237b640 4 set_mon_vals no callback set 2024-03-29T11:21:34.292 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.312+0000 7f4d1affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:34.292 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.312+0000 7f4d1affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:34.293 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.312+0000 7f4d1affd640 10 monclient: dump: 2024-03-29T11:21:34.293 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:21:34.293 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:21:34.293 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:34.293 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:34.293 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:34.293 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:21:34.293 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:21:34.293 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:21:34.293 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:21:34.293 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:21:34.293 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.313+0000 7f4d2237b640 10 monclient: shutdown 2024-03-29T11:21:34.293 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.313+0000 7f4d2237b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:34.294 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.313+0000 7f4d2237b640 1 -- 172.21.15.47:0/3637869367 >> v1:172.21.15.121:6789/0 conn(0x7f4d1c0b17d0 legacy=0x7f4d1c0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:34.294 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.313+0000 7f4d2237b640 1 -- 172.21.15.47:0/3637869367 shutdown_connections 2024-03-29T11:21:34.294 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.313+0000 7f4d2237b640 1 -- 172.21.15.47:0/3637869367 >> 172.21.15.47:0/3637869367 conn(0x7f4d1c066a00 msgr2=0x7f4d1c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:34.294 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.313+0000 7f4d2237b640 1 -- 172.21.15.47:0/3637869367 shutdown_connections 2024-03-29T11:21:34.294 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.313+0000 7f4d2237b640 1 -- 172.21.15.47:0/3637869367 wait complete. 2024-03-29T11:21:34.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.314+0000 7f4d2237b640 10 monclient: build_initial_monmap 2024-03-29T11:21:34.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.314+0000 7f4d2237b640 10 monclient: monmap: 2024-03-29T11:21:34.301 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:21:34.302 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:34.302 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:21:34.302 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:21:34.302 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:34.302 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:21:34.302 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:21:34.302 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:21:34.302 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:21:34.302 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:21:34.302 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.314+0000 7f4d2237b640 1 librados: starting msgr at 2024-03-29T11:21:34.302 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.314+0000 7f4d2237b640 1 librados: starting objecter 2024-03-29T11:21:34.302 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.314+0000 7f4d2237b640 1 Processor -- start 2024-03-29T11:21:34.302 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.315+0000 7f4d2237b640 1 -- start start 2024-03-29T11:21:34.302 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.315+0000 7f4d2237b640 1 librados: setting wanted keys 2024-03-29T11:21:34.303 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.315+0000 7f4d2237b640 1 librados: calling monclient init 2024-03-29T11:21:34.303 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.315+0000 7f4d2237b640 10 monclient: init 2024-03-29T11:21:34.303 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.315+0000 7f4d2237b640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:34.303 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.316+0000 7f4d2237b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:21:34.303 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.316+0000 7f4d2237b640 10 monclient(hunting): picked mon.noname-c con 0x7f4d1c0b17d0 addr v1:172.21.15.47:6790/0 2024-03-29T11:21:34.303 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.316+0000 7f4d2237b640 10 monclient(hunting): picked mon.noname-a con 0x7f4d1c1422b0 addr v1:172.21.15.47:6789/0 2024-03-29T11:21:34.303 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.316+0000 7f4d1bfff640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f4d1c0b17d0 0x7f4d1c0af500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:60760/0 (socket says 172.21.15.47:60760) 2024-03-29T11:21:34.303 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.316+0000 7f4d1b7fe640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f4d1c1422b0 0x7f4d1c12d950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:38386/0 (socket says 172.21.15.47:38386) 2024-03-29T11:21:34.303 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.316+0000 7f4d1bfff640 1 -- 172.21.15.47:0/3531034208 learned_addr learned my addr 172.21.15.47:0/3531034208 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:21:34.303 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.316+0000 7f4d2237b640 10 monclient(hunting): picked mon.noname-b con 0x7f4d1c142690 addr v1:172.21.15.121:6789/0 2024-03-29T11:21:34.303 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.316+0000 7f4d18ff9640 10 client.?.objecter ms_handle_connect 0x7f4d1c0b17d0 2024-03-29T11:21:34.303 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.316+0000 7f4d2237b640 1 -- 172.21.15.47:0/3531034208 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d1c0aeb20 con 0x7f4d1c1422b0 2024-03-29T11:21:34.303 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.316+0000 7f4d2237b640 1 -- 172.21.15.47:0/3531034208 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d1c0aee00 con 0x7f4d1c142690 2024-03-29T11:21:34.304 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.316+0000 7f4d18ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:34.304 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.316+0000 7f4d18ff9640 10 client.?.objecter ms_handle_connect 0x7f4d1c142690 2024-03-29T11:21:34.304 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.316+0000 7f4d18ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:34.304 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.316+0000 7f4d18ff9640 10 client.?.objecter ms_handle_connect 0x7f4d1c1422b0 2024-03-29T11:21:34.304 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.316+0000 7f4d18ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:34.304 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.316+0000 7f4d2237b640 1 -- 172.21.15.47:0/3531034208 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d1c0af0e0 con 0x7f4d1c0b17d0 2024-03-29T11:21:34.304 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.316+0000 7f4d2237b640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:34.304 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.316+0000 7f4d2237b640 10 monclient(hunting): authenticate will time out at 1337.504770s 2024-03-29T11:21:34.304 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.317+0000 7f4d18ff9640 1 -- 172.21.15.47:0/3531034208 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2737534132 0 0) 0x7f4d1c0aeb20 con 0x7f4d1c1422b0 2024-03-29T11:21:34.304 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.317+0000 7f4d18ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:21:34.304 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.317+0000 7f4d18ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:34.304 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.317+0000 7f4d18ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:34.304 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.317+0000 7f4d18ff9640 10 monclient(hunting): my global_id is 4518 2024-03-29T11:21:34.304 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.317+0000 7f4d18ff9640 1 -- 172.21.15.47:0/3531034208 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d08004eb0 con 0x7f4d1c1422b0 2024-03-29T11:21:34.305 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.317+0000 7f4d18ff9640 1 -- 172.21.15.47:0/3531034208 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2497979277 0 0) 0x7f4d1c0aee00 con 0x7f4d1c142690 2024-03-29T11:21:34.305 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.317+0000 7f4d18ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:21:34.305 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.317+0000 7f4d18ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:34.305 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.317+0000 7f4d18ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:34.305 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.317+0000 7f4d18ff9640 10 monclient(hunting): my global_id is 4507 2024-03-29T11:21:34.305 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.317+0000 7f4d18ff9640 1 -- 172.21.15.47:0/3531034208 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d1c0aeb20 con 0x7f4d1c142690 2024-03-29T11:21:34.305 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.317+0000 7f4d18ff9640 1 -- 172.21.15.47:0/3531034208 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1018555658 0 0) 0x7f4d1c0af0e0 con 0x7f4d1c0b17d0 2024-03-29T11:21:34.305 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.317+0000 7f4d18ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:21:34.305 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.317+0000 7f4d18ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:34.305 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.317+0000 7f4d18ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:34.305 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.317+0000 7f4d18ff9640 10 monclient(hunting): my global_id is 4580 2024-03-29T11:21:34.305 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.317+0000 7f4d18ff9640 1 -- 172.21.15.47:0/3531034208 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d1c0aee00 con 0x7f4d1c0b17d0 2024-03-29T11:21:34.305 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.317+0000 7f4d18ff9640 1 -- 172.21.15.47:0/3531034208 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2334299065 0 0) 0x7f4d1c0aeb20 con 0x7f4d1c142690 2024-03-29T11:21:34.305 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.318+0000 7f4d18ff9640 1 -- 172.21.15.47:0/3531034208 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d1c0af0e0 con 0x7f4d1c142690 2024-03-29T11:21:34.306 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.318+0000 7f4d18ff9640 1 -- 172.21.15.47:0/3531034208 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f4d0c001960 con 0x7f4d1c142690 2024-03-29T11:21:34.306 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.318+0000 7f4d18ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:34.306 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.318+0000 7f4d18ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:34.306 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.318+0000 7f4d18ff9640 10 monclient(hunting): dump: 2024-03-29T11:21:34.306 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:21:34.306 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:21:34.306 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:34.306 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:34.306 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:34.306 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:21:34.306 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:21:34.306 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:21:34.306 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:21:34.306 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:21:34.307 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.318+0000 7f4d18ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:34.307 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.318+0000 7f4d18ff9640 1 -- 172.21.15.47:0/3531034208 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3440349878 0 0) 0x7f4d1c0aee00 con 0x7f4d1c0b17d0 2024-03-29T11:21:34.307 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.318+0000 7f4d18ff9640 1 -- 172.21.15.47:0/3531034208 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d1c0aeb20 con 0x7f4d1c0b17d0 2024-03-29T11:21:34.307 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.318+0000 7f4d18ff9640 1 -- 172.21.15.47:0/3531034208 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f4d14001cb0 con 0x7f4d1c0b17d0 2024-03-29T11:21:34.307 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.318+0000 7f4d18ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:34.307 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.318+0000 7f4d18ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:34.307 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.318+0000 7f4d18ff9640 10 monclient(hunting): dump: 2024-03-29T11:21:34.307 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:21:34.307 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:21:34.307 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:34.307 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:34.307 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:34.307 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:21:34.307 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:21:34.308 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:21:34.308 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:21:34.308 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:21:34.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.318+0000 7f4d18ff9640 1 -- 172.21.15.47:0/3531034208 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1148682290 0 0) 0x7f4d1c0aeb20 con 0x7f4d1c0b17d0 2024-03-29T11:21:34.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.318+0000 7f4d18ff9640 1 -- 172.21.15.47:0/3531034208 >> v1:172.21.15.121:6789/0 conn(0x7f4d1c142690 legacy=0x7f4d1c12e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:34.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.318+0000 7f4d18ff9640 1 -- 172.21.15.47:0/3531034208 >> v1:172.21.15.47:6789/0 conn(0x7f4d1c1422b0 legacy=0x7f4d1c12d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:34.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.318+0000 7f4d18ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:21:34.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.318+0000 7f4d18ff9640 1 monclient: found mon.c 2024-03-29T11:21:34.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.318+0000 7f4d18ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:34.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.318+0000 7f4d18ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:21:34.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.318+0000 7f4d18ff9640 1 -- 172.21.15.47:0/3531034208 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d1c0ad6d0 con 0x7f4d1c0b17d0 2024-03-29T11:21:34.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.318+0000 7f4d18ff9640 10 monclient: _finish_auth 0 2024-03-29T11:21:34.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.318+0000 7f4d18ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:21:34.309 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.318+0000 7f4d18ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:34.309 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.318+0000 7f4d2237b640 5 monclient: authenticate success, global_id 4580 2024-03-29T11:21:34.309 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.318+0000 7f4d2237b640 10 monclient: _renew_subs 2024-03-29T11:21:34.309 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.318+0000 7f4d2237b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:21:34.309 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.318+0000 7f4d2237b640 1 -- 172.21.15.47:0/3531034208 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d1c0afc10 con 0x7f4d1c0b17d0 2024-03-29T11:21:34.309 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.319+0000 7f4d2237b640 10 client.4580.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:34.309 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.319+0000 7f4d2237b640 10 monclient: _renew_subs 2024-03-29T11:21:34.309 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.319+0000 7f4d2237b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:21:34.309 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.319+0000 7f4d2237b640 1 -- 172.21.15.47:0/3531034208 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d1c0addd0 con 0x7f4d1c0b17d0 2024-03-29T11:21:34.309 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.319+0000 7f4d2237b640 1 librados: init done 2024-03-29T11:21:34.309 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.319+0000 7f4d18ff9640 1 -- 172.21.15.47:0/3531034208 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4d140020c0 con 0x7f4d1c0b17d0 2024-03-29T11:21:34.309 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.319+0000 7f4d18ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:34.309 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.319+0000 7f4d18ff9640 1 -- 172.21.15.47:0/3531034208 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f4d14004d00 con 0x7f4d1c0b17d0 2024-03-29T11:21:34.309 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.319+0000 7f4d1a7fc640 4 set_mon_vals no callback set 2024-03-29T11:21:34.310 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.319+0000 7f4d18ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:34.310 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.319+0000 7f4d18ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:34.310 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.319+0000 7f4d18ff9640 10 monclient: dump: 2024-03-29T11:21:34.310 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:21:34.310 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:21:34.310 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:34.310 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:34.310 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:34.310 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:21:34.310 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:21:34.310 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:21:34.310 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:21:34.310 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:21:34.310 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.321+0000 7f4d2237b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:34.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.321+0000 7f4d2237b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:34.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.321+0000 7f4d2237b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:21:34.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.321+0000 7f4d2237b640 1 -- 172.21.15.47:0/3531034208 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cfc005080 con 0x7f4d1c0b17d0 2024-03-29T11:21:34.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.322+0000 7f4d18ff9640 1 -- 172.21.15.47:0/3531034208 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7f4d14004e60 con 0x7f4d1c0b17d0 2024-03-29T11:21:34.312 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.324+0000 7f4d1b7fe640 10 monclient: get_auth_request con 0x7f4d0806df20 auth_method 0 2024-03-29T11:21:34.312 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.324+0000 7f4d18ff9640 1 -- 172.21.15.47:0/3531034208 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7f4d14084dc0 con 0x7f4d1c0b17d0 2024-03-29T11:21:34.312 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.324+0000 7f4d18ff9640 10 client.4580.objecter ms_dispatch 0x7f4d1c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:34.312 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.324+0000 7f4d18ff9640 3 client.4580.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:34.312 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.324+0000 7f4d18ff9640 3 client.4580.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:34.312 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.324+0000 7f4d18ff9640 20 client.4580.objecter dump_active .. 0 homeless 2024-03-29T11:21:34.312 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.324+0000 7f4d18ff9640 10 client.4580.objecter ms_handle_connect 0x7f4d0806df20 2024-03-29T11:21:34.312 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.332+0000 7f4d18ff9640 1 -- 172.21.15.47:0/3531034208 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4d14050700 con 0x7f4d1c0b17d0 2024-03-29T11:21:34.313 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.332+0000 7f4d18ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:34.313 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.332+0000 7f4d18ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:34.709 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.732+0000 7f4d2237b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:34.709 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.732+0000 7f4d2237b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:34.709 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.732+0000 7f4d2237b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:21:34.709 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.732+0000 7f4d2237b640 1 -- 172.21.15.47:0/3531034208 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4cfc005330 con 0x7f4d1c0b17d0 2024-03-29T11:21:34.710 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.733+0000 7f4d18ff9640 1 -- 172.21.15.47:0/3531034208 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7f4d14052630 con 0x7f4d1c0b17d0 2024-03-29T11:21:34.710 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.733+0000 7f4d18ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:34.710 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.733+0000 7f4d18ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:34.710 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:21:34.710 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:34.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.737+0000 7f4d2237b640 10 librados: watch_flush enter 2024-03-29T11:21:34.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.737+0000 7f4d2237b640 10 librados: watch_flush exit 2024-03-29T11:21:34.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.737+0000 7f4d2237b640 20 client.4580.objecter shutdown clearing up homeless session... 2024-03-29T11:21:34.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.737+0000 7f4d2237b640 10 client.4580.objecter successfully canceled tick 2024-03-29T11:21:34.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.737+0000 7f4d2237b640 1 -- 172.21.15.47:0/3531034208 >> v1:172.21.15.121:6816/1964117370 conn(0x7f4d0806df20 legacy=0x7f4d080703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:34.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.737+0000 7f4d2237b640 10 monclient: shutdown 2024-03-29T11:21:34.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.737+0000 7f4d2237b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:34.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.737+0000 7f4d2237b640 1 -- 172.21.15.47:0/3531034208 >> v1:172.21.15.47:6790/0 conn(0x7f4d1c0b17d0 legacy=0x7f4d1c0af500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:34.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.737+0000 7f4d2237b640 1 -- 172.21.15.47:0/3531034208 shutdown_connections 2024-03-29T11:21:34.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.737+0000 7f4d2237b640 1 -- 172.21.15.47:0/3531034208 >> 172.21.15.47:0/3531034208 conn(0x7f4d1c066a00 msgr2=0x7f4d1c133b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:34.715 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.737+0000 7f4d2237b640 1 -- 172.21.15.47:0/3531034208 shutdown_connections 2024-03-29T11:21:34.715 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.737+0000 7f4d2237b640 1 -- 172.21.15.47:0/3531034208 wait complete. 2024-03-29T11:21:34.715 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:34.738+0000 7f4d2237b640 1 librados: shutdown 2024-03-29T11:21:34.730 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:41.537 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:41.694 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.717+0000 7f37abfff640 10 monclient: get_monmap_and_config 2024-03-29T11:21:41.694 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.717+0000 7f37abfff640 10 monclient: build_initial_monmap 2024-03-29T11:21:41.694 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.717+0000 7f37abfff640 10 monclient: monmap: 2024-03-29T11:21:41.694 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:21:41.694 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:21:41.694 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:21:41.718532+0000 2024-03-29T11:21:41.694 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:21:41.718532+0000 2024-03-29T11:21:41.694 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:41.695 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:21:41.695 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:21:41.695 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:21:41.695 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:21:41.695 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:21:41.696 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.719+0000 7f37abfff640 1 Processor -- start 2024-03-29T11:21:41.696 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.719+0000 7f37abfff640 1 -- start start 2024-03-29T11:21:41.696 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.719+0000 7f37abfff640 10 monclient: init 2024-03-29T11:21:41.697 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.720+0000 7f37abfff640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:41.697 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.720+0000 7f37abfff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:21:41.697 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.720+0000 7f37abfff640 10 monclient(hunting): picked mon.noname-c con 0x7f37ac0b2790 addr v1:172.21.15.121:6789/0 2024-03-29T11:21:41.697 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.720+0000 7f37abfff640 10 monclient(hunting): picked mon.noname-b con 0x7f37ac143c80 addr v1:172.21.15.47:6790/0 2024-03-29T11:21:41.697 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.720+0000 7f37abfff640 10 monclient(hunting): picked mon.noname-a con 0x7f37ac146e30 addr v1:172.21.15.47:6789/0 2024-03-29T11:21:41.697 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.720+0000 7f37abfff640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37ac146290 con 0x7f37ac146e30 2024-03-29T11:21:41.697 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.720+0000 7f37abfff640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37ac0b3970 con 0x7f37ac0b2790 2024-03-29T11:21:41.697 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.720+0000 7f37abfff640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37ac06ee80 con 0x7f37ac143c80 2024-03-29T11:21:41.697 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.720+0000 7f37abfff640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:41.697 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.720+0000 7f37abfff640 10 monclient(hunting): authenticate will time out at 1344.908641s 2024-03-29T11:21:41.703 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.726+0000 7f37aa7fc640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f37ac143c80 0x7f37ac146720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:45215/0 (socket says 172.21.15.47:45215) 2024-03-29T11:21:41.703 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.726+0000 7f37aa7fc640 1 -- 172.21.15.47:0/2069649692 learned_addr learned my addr 172.21.15.47:0/2069649692 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:21:41.704 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.728+0000 7f37a9ffb640 1 -- 172.21.15.47:0/2069649692 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2277692871 0 0) 0x7f37ac06ee80 con 0x7f37ac143c80 2024-03-29T11:21:41.704 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.728+0000 7f37a9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:41.704 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.728+0000 7f37a9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:41.704 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.728+0000 7f37a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:41.704 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.728+0000 7f37a9ffb640 10 monclient(hunting): my global_id is 4583 2024-03-29T11:21:41.704 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.728+0000 7f37a9ffb640 1 -- 172.21.15.47:0/2069649692 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3794003520 con 0x7f37ac143c80 2024-03-29T11:21:41.705 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.728+0000 7f37a9ffb640 1 -- 172.21.15.47:0/2069649692 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2774226253 0 0) 0x7f37ac146290 con 0x7f37ac146e30 2024-03-29T11:21:41.705 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.728+0000 7f37a9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:41.705 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.728+0000 7f37a9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:41.706 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.728+0000 7f37a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:41.706 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.728+0000 7f37a9ffb640 10 monclient(hunting): my global_id is 4521 2024-03-29T11:21:41.706 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.728+0000 7f37a9ffb640 1 -- 172.21.15.47:0/2069649692 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37ac06ee80 con 0x7f37ac146e30 2024-03-29T11:21:41.706 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.728+0000 7f37a9ffb640 1 -- 172.21.15.47:0/2069649692 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2862736351 0 0) 0x7f3794003520 con 0x7f37ac143c80 2024-03-29T11:21:41.706 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.728+0000 7f37a9ffb640 1 -- 172.21.15.47:0/2069649692 >> v1:172.21.15.121:6789/0 conn(0x7f37ac0b2790 legacy=0x7f37ac143570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:41.706 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.728+0000 7f37a9ffb640 1 -- 172.21.15.47:0/2069649692 >> v1:172.21.15.47:6789/0 conn(0x7f37ac146e30 legacy=0x7f37ac151bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:41.706 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.728+0000 7f37a9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:21:41.706 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.728+0000 7f37a9ffb640 1 monclient: found mon.noname-b 2024-03-29T11:21:41.706 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.728+0000 7f37a9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:41.706 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.728+0000 7f37a9ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-29T11:21:41.706 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.728+0000 7f37a9ffb640 1 -- 172.21.15.47:0/2069649692 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37ac06f160 con 0x7f37ac143c80 2024-03-29T11:21:41.706 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.728+0000 7f37a9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:21:41.706 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.728+0000 7f37a9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:21:41.706 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.728+0000 7f37a9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:41.706 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.728+0000 7f37a9ffb640 1 -- 172.21.15.47:0/2069649692 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f37a0001690 con 0x7f37ac143c80 2024-03-29T11:21:41.707 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.728+0000 7f37a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:41.707 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.728+0000 7f37a9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:41.707 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.728+0000 7f37a9ffb640 10 monclient: dump: 2024-03-29T11:21:41.707 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:21:41.707 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:21:41.707 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:41.707 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:41.707 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:41.707 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:21:41.707 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:21:41.707 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:21:41.707 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:21:41.707 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:21:41.707 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.729+0000 7f37a9ffb640 1 -- 172.21.15.47:0/2069649692 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f37a00017f0 con 0x7f37ac143c80 2024-03-29T11:21:41.708 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.729+0000 7f37a9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:41.708 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.729+0000 7f37a9ffb640 1 -- 172.21.15.47:0/2069649692 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f37a0001c00 con 0x7f37ac143c80 2024-03-29T11:21:41.708 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.729+0000 7f37a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:41.708 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.729+0000 7f37a9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:41.708 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.729+0000 7f37a9ffb640 10 monclient: dump: 2024-03-29T11:21:41.708 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:21:41.708 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:21:41.708 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:41.708 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:41.708 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:41.708 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:21:41.708 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:21:41.708 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:21:41.708 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:21:41.708 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:21:41.709 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.729+0000 7f37abfff640 5 monclient: authenticate success, global_id 4583 2024-03-29T11:21:41.709 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.729+0000 7f37abfff640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:41.709 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.729+0000 7f37abfff640 4 set_mon_vals no callback set 2024-03-29T11:21:41.709 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.730+0000 7f37abfff640 10 monclient: shutdown 2024-03-29T11:21:41.709 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.730+0000 7f37abfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:41.709 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.731+0000 7f37abfff640 1 -- 172.21.15.47:0/2069649692 >> v1:172.21.15.47:6790/0 conn(0x7f37ac143c80 legacy=0x7f37ac146720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:41.709 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.731+0000 7f37abfff640 1 -- 172.21.15.47:0/2069649692 shutdown_connections 2024-03-29T11:21:41.709 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.731+0000 7f37abfff640 1 -- 172.21.15.47:0/2069649692 >> 172.21.15.47:0/2069649692 conn(0x7f37ac06d070 msgr2=0x7f37ac066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:41.710 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.733+0000 7f37abfff640 1 -- 172.21.15.47:0/2069649692 shutdown_connections 2024-03-29T11:21:41.711 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.734+0000 7f37abfff640 1 -- 172.21.15.47:0/2069649692 wait complete. 2024-03-29T11:21:41.711 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.734+0000 7f37abfff640 10 monclient: build_initial_monmap 2024-03-29T11:21:41.711 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.734+0000 7f37abfff640 10 monclient: monmap: 2024-03-29T11:21:41.711 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:21:41.711 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:41.711 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:21:41.711 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:21:41.711 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:41.711 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:21:41.711 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:21:41.711 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:21:41.711 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:21:41.711 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:21:41.713 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.735+0000 7f37abfff640 1 librados: starting msgr at 2024-03-29T11:21:41.713 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.735+0000 7f37abfff640 1 librados: starting objecter 2024-03-29T11:21:41.713 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.735+0000 7f37abfff640 1 Processor -- start 2024-03-29T11:21:41.713 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.735+0000 7f37abfff640 1 -- start start 2024-03-29T11:21:41.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.735+0000 7f37abfff640 1 librados: setting wanted keys 2024-03-29T11:21:41.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.735+0000 7f37abfff640 1 librados: calling monclient init 2024-03-29T11:21:41.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.735+0000 7f37abfff640 10 monclient: init 2024-03-29T11:21:41.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.736+0000 7f37abfff640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:41.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.736+0000 7f37abfff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:21:41.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.736+0000 7f37abfff640 10 monclient(hunting): picked mon.noname-a con 0x7f37ac0b2790 addr v1:172.21.15.47:6789/0 2024-03-29T11:21:41.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.736+0000 7f37abfff640 10 monclient(hunting): picked mon.noname-c con 0x7f37ac146e30 addr v1:172.21.15.47:6790/0 2024-03-29T11:21:41.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.736+0000 7f37abfff640 10 monclient(hunting): picked mon.noname-b con 0x7f37ac140010 addr v1:172.21.15.121:6789/0 2024-03-29T11:21:41.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.736+0000 7f37abfff640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37ac0b6e00 con 0x7f37ac0b2790 2024-03-29T11:21:41.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.736+0000 7f37abfff640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37ac0b6340 con 0x7f37ac140010 2024-03-29T11:21:41.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.736+0000 7f37abfff640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37ac0b7050 con 0x7f37ac146e30 2024-03-29T11:21:41.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.736+0000 7f37abfff640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:41.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.736+0000 7f37abfff640 10 monclient(hunting): authenticate will time out at 1344.924467s 2024-03-29T11:21:41.715 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.739+0000 7f37ab7fe640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f37ac140010 0x7f37ac1e6e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.47:40321/0 (socket says 172.21.15.47:40321) 2024-03-29T11:21:41.715 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.739+0000 7f37ab7fe640 1 -- 172.21.15.47:0/1810974345 learned_addr learned my addr 172.21.15.47:0/1810974345 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:21:41.723 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.743+0000 7f37937fe640 10 client.?.objecter ms_handle_connect 0x7f37ac140010 2024-03-29T11:21:41.723 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.743+0000 7f37937fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:41.723 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.743+0000 7f37937fe640 10 client.?.objecter ms_handle_connect 0x7f37ac0b2790 2024-03-29T11:21:41.723 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.743+0000 7f37937fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:41.723 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.743+0000 7f37937fe640 1 -- 172.21.15.47:0/1810974345 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3467804514 0 0) 0x7f37ac0b6340 con 0x7f37ac140010 2024-03-29T11:21:41.723 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.743+0000 7f37937fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:41.723 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.743+0000 7f37937fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:41.724 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.743+0000 7f37937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:41.724 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.743+0000 7f37937fe640 10 monclient(hunting): my global_id is 4513 2024-03-29T11:21:41.724 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.743+0000 7f37937fe640 1 -- 172.21.15.47:0/1810974345 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3798004eb0 con 0x7f37ac140010 2024-03-29T11:21:41.724 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.743+0000 7f37937fe640 1 -- 172.21.15.47:0/1810974345 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3656605308 0 0) 0x7f37ac0b6e00 con 0x7f37ac0b2790 2024-03-29T11:21:41.724 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.743+0000 7f37937fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:41.724 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.743+0000 7f37937fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:41.724 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.743+0000 7f37937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:41.724 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.744+0000 7f37937fe640 10 monclient(hunting): my global_id is 4524 2024-03-29T11:21:41.724 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.744+0000 7f37937fe640 1 -- 172.21.15.47:0/1810974345 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37ac0b6340 con 0x7f37ac0b2790 2024-03-29T11:21:41.724 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.744+0000 7f37937fe640 10 client.?.objecter ms_handle_connect 0x7f37ac146e30 2024-03-29T11:21:41.724 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.744+0000 7f37937fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:41.724 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.744+0000 7f37937fe640 1 -- 172.21.15.47:0/1810974345 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2091173819 0 0) 0x7f3798004eb0 con 0x7f37ac140010 2024-03-29T11:21:41.724 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.744+0000 7f37937fe640 1 -- 172.21.15.47:0/1810974345 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f37ac0b6e00 con 0x7f37ac140010 2024-03-29T11:21:41.724 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.744+0000 7f37937fe640 1 -- 172.21.15.47:0/1810974345 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f379c001910 con 0x7f37ac140010 2024-03-29T11:21:41.724 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.744+0000 7f37937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:41.724 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.744+0000 7f37937fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:41.725 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.744+0000 7f37937fe640 10 monclient(hunting): dump: 2024-03-29T11:21:41.725 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:21:41.725 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:21:41.725 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:41.725 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:41.725 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:41.725 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:21:41.725 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:21:41.725 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:21:41.725 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:21:41.725 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:21:41.725 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.744+0000 7f37937fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:41.725 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.744+0000 7f37937fe640 1 -- 172.21.15.47:0/1810974345 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 590312828 0 0) 0x7f37ac0b7050 con 0x7f37ac146e30 2024-03-29T11:21:41.725 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.744+0000 7f37937fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:41.725 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.744+0000 7f37937fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:41.725 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.744+0000 7f37937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:41.726 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.744+0000 7f37937fe640 10 monclient(hunting): my global_id is 4586 2024-03-29T11:21:41.726 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.744+0000 7f37937fe640 1 -- 172.21.15.47:0/1810974345 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3798004eb0 con 0x7f37ac146e30 2024-03-29T11:21:41.726 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.744+0000 7f37937fe640 1 -- 172.21.15.47:0/1810974345 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 273681796 0 0) 0x7f37ac0b6e00 con 0x7f37ac140010 2024-03-29T11:21:41.726 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.744+0000 7f37937fe640 1 -- 172.21.15.47:0/1810974345 >> v1:172.21.15.47:6790/0 conn(0x7f37ac146e30 legacy=0x7f37ac135e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:41.726 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.744+0000 7f37937fe640 1 -- 172.21.15.47:0/1810974345 >> v1:172.21.15.47:6789/0 conn(0x7f37ac0b2790 legacy=0x7f37ac0b5c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:41.726 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.744+0000 7f37937fe640 10 monclient: _finish_hunting 0 2024-03-29T11:21:41.726 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.744+0000 7f37937fe640 1 monclient: found mon.b 2024-03-29T11:21:41.726 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.744+0000 7f37937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:41.726 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.744+0000 7f37937fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:21:41.726 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.744+0000 7f37937fe640 1 -- 172.21.15.47:0/1810974345 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37ac0b7330 con 0x7f37ac140010 2024-03-29T11:21:41.726 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.744+0000 7f37937fe640 10 monclient: _finish_auth 0 2024-03-29T11:21:41.726 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.744+0000 7f37937fe640 10 monclient: _check_auth_tickets 2024-03-29T11:21:41.726 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.744+0000 7f37937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:41.726 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.744+0000 7f37abfff640 5 monclient: authenticate success, global_id 4513 2024-03-29T11:21:41.727 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.744+0000 7f37abfff640 10 monclient: _renew_subs 2024-03-29T11:21:41.727 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.744+0000 7f37abfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:21:41.727 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.744+0000 7f37abfff640 1 -- 172.21.15.47:0/1810974345 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37ac0b7610 con 0x7f37ac140010 2024-03-29T11:21:41.727 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.745+0000 7f37abfff640 10 client.4513.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:41.727 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.745+0000 7f37abfff640 10 monclient: _renew_subs 2024-03-29T11:21:41.727 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.745+0000 7f37abfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:21:41.727 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.745+0000 7f37abfff640 1 -- 172.21.15.47:0/1810974345 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f37ac145ea0 con 0x7f37ac140010 2024-03-29T11:21:41.727 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.745+0000 7f37abfff640 1 librados: init done 2024-03-29T11:21:41.727 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.745+0000 7f37937fe640 1 -- 172.21.15.47:0/1810974345 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f379c001a70 con 0x7f37ac140010 2024-03-29T11:21:41.727 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.745+0000 7f37937fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:41.727 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.745+0000 7f37937fe640 1 -- 172.21.15.47:0/1810974345 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f379c003870 con 0x7f37ac140010 2024-03-29T11:21:41.727 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.745+0000 7f37937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:41.727 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.745+0000 7f37937fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:41.727 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.745+0000 7f37937fe640 10 monclient: dump: 2024-03-29T11:21:41.727 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:21:41.727 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:21:41.728 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:41.728 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:41.728 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:41.728 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:21:41.728 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:21:41.728 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:21:41.728 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:21:41.728 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:21:41.728 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.747+0000 7f37a97fa640 4 set_mon_vals no callback set 2024-03-29T11:21:41.729 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.747+0000 7f37937fe640 1 -- 172.21.15.47:0/1810974345 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7f379c0039d0 con 0x7f37ac140010 2024-03-29T11:21:41.729 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.748+0000 7f37937fe640 1 -- 172.21.15.47:0/1810974345 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7f379c084a00 con 0x7f37ac140010 2024-03-29T11:21:41.729 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.748+0000 7f37937fe640 10 client.4513.objecter ms_dispatch 0x7f37ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:41.729 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.748+0000 7f37937fe640 3 client.4513.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:41.729 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.748+0000 7f37937fe640 3 client.4513.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:41.730 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.748+0000 7f37937fe640 20 client.4513.objecter dump_active .. 0 homeless 2024-03-29T11:21:41.730 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.748+0000 7f37aaffd640 10 monclient: get_auth_request con 0x7f379806dd30 auth_method 0 2024-03-29T11:21:41.730 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.749+0000 7f37abfff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:41.730 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.749+0000 7f37937fe640 10 client.4513.objecter ms_handle_connect 0x7f379806dd30 2024-03-29T11:21:41.730 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.749+0000 7f37abfff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:41.730 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.749+0000 7f37abfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:21:41.730 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.750+0000 7f37abfff640 1 -- 172.21.15.47:0/1810974345 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f378c005080 con 0x7f37ac140010 2024-03-29T11:21:41.734 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.758+0000 7f37937fe640 1 -- 172.21.15.47:0/1810974345 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f379c050110 con 0x7f37ac140010 2024-03-29T11:21:41.734 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.758+0000 7f37937fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:41.734 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:41.758+0000 7f37937fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:42.074 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:42.097+0000 7f37abfff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:42.074 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:42.097+0000 7f37abfff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:42.074 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:42.097+0000 7f37abfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:21:42.074 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:42.097+0000 7f37abfff640 1 -- 172.21.15.47:0/1810974345 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f378c005330 con 0x7f37ac140010 2024-03-29T11:21:42.076 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:42.099+0000 7f37937fe640 1 -- 172.21.15.47:0/1810974345 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7f379c052210 con 0x7f37ac140010 2024-03-29T11:21:42.076 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:42.099+0000 7f37937fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:42.076 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:42.099+0000 7f37937fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:42.077 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:21:42.077 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:42.082 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:42.104+0000 7f37abfff640 10 librados: watch_flush enter 2024-03-29T11:21:42.082 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:42.104+0000 7f37abfff640 10 librados: watch_flush exit 2024-03-29T11:21:42.082 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:42.104+0000 7f37abfff640 20 client.4513.objecter shutdown clearing up homeless session... 2024-03-29T11:21:42.082 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:42.104+0000 7f37abfff640 10 client.4513.objecter successfully canceled tick 2024-03-29T11:21:42.082 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:42.104+0000 7f37abfff640 1 -- 172.21.15.47:0/1810974345 >> v1:172.21.15.121:6816/1964117370 conn(0x7f379806dd30 legacy=0x7f37980701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:42.082 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:42.104+0000 7f37abfff640 10 monclient: shutdown 2024-03-29T11:21:42.082 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:42.104+0000 7f37abfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:42.082 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:42.104+0000 7f37abfff640 1 -- 172.21.15.47:0/1810974345 >> v1:172.21.15.121:6789/0 conn(0x7f37ac140010 legacy=0x7f37ac1e6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:42.084 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:42.107+0000 7f37abfff640 1 -- 172.21.15.47:0/1810974345 shutdown_connections 2024-03-29T11:21:42.084 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:42.107+0000 7f37abfff640 1 -- 172.21.15.47:0/1810974345 >> 172.21.15.47:0/1810974345 conn(0x7f37ac06d070 msgr2=0x7f37ac069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:42.084 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:42.107+0000 7f37abfff640 1 -- 172.21.15.47:0/1810974345 shutdown_connections 2024-03-29T11:21:42.084 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:42.107+0000 7f37abfff640 1 -- 172.21.15.47:0/1810974345 wait complete. 2024-03-29T11:21:42.084 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:42.107+0000 7f37abfff640 1 librados: shutdown 2024-03-29T11:21:42.102 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:48.908 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:49.057 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.079+0000 7f20a2367640 10 monclient: get_monmap_and_config 2024-03-29T11:21:49.058 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.079+0000 7f20a2367640 10 monclient: build_initial_monmap 2024-03-29T11:21:49.058 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.079+0000 7f20a2367640 10 monclient: monmap: 2024-03-29T11:21:49.058 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:21:49.058 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:21:49.058 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:21:49.080067+0000 2024-03-29T11:21:49.058 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:21:49.080067+0000 2024-03-29T11:21:49.058 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:49.058 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:21:49.058 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:21:49.058 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:21:49.058 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:21:49.058 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:21:49.060 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.083+0000 7f20a2367640 1 Processor -- start 2024-03-29T11:21:49.061 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.084+0000 7f20a2367640 1 -- start start 2024-03-29T11:21:49.061 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.084+0000 7f20a2367640 10 monclient: init 2024-03-29T11:21:49.062 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.084+0000 7f20a2367640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:49.062 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.084+0000 7f20a2367640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:21:49.062 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.084+0000 7f20a2367640 10 monclient(hunting): picked mon.noname-c con 0x7f209c0a9ac0 addr v1:172.21.15.121:6789/0 2024-03-29T11:21:49.062 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.084+0000 7f20a2367640 10 monclient(hunting): picked mon.noname-b con 0x7f209c13ee90 addr v1:172.21.15.47:6790/0 2024-03-29T11:21:49.062 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.085+0000 7f20a2367640 10 monclient(hunting): picked mon.noname-a con 0x7f209c0af900 addr v1:172.21.15.47:6789/0 2024-03-29T11:21:49.062 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.085+0000 7f20a2367640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f209c0afd10 con 0x7f209c0af900 2024-03-29T11:21:49.062 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.085+0000 7f20a2367640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f209c0af6f0 con 0x7f209c0a9ac0 2024-03-29T11:21:49.062 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.085+0000 7f20a2367640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f209c0aac80 con 0x7f209c13ee90 2024-03-29T11:21:49.062 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.085+0000 7f20a2367640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:49.062 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.085+0000 7f20a2367640 10 monclient(hunting): authenticate will time out at 1352.273194s 2024-03-29T11:21:49.064 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.088+0000 7f20a0904640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f209c0af900 0x7f209c149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:57701/0 (socket says 172.21.15.47:57701) 2024-03-29T11:21:49.064 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.088+0000 7f20a0904640 1 -- 172.21.15.47:0/695548842 learned_addr learned my addr 172.21.15.47:0/695548842 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:21:49.067 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.088+0000 7f209affd640 1 -- 172.21.15.47:0/695548842 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1220484438 0 0) 0x7f209c0afd10 con 0x7f209c0af900 2024-03-29T11:21:49.067 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.088+0000 7f209affd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:49.067 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.088+0000 7f209affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:49.067 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.088+0000 7f209affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:49.067 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.088+0000 7f209affd640 10 monclient(hunting): my global_id is 4527 2024-03-29T11:21:49.067 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.088+0000 7f209affd640 1 -- 172.21.15.47:0/695548842 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2084003520 con 0x7f209c0af900 2024-03-29T11:21:49.067 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.088+0000 7f209affd640 1 -- 172.21.15.47:0/695548842 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3163098503 0 0) 0x7f2084003520 con 0x7f209c0af900 2024-03-29T11:21:49.067 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.089+0000 7f209affd640 1 -- 172.21.15.47:0/695548842 >> v1:172.21.15.47:6790/0 conn(0x7f209c13ee90 legacy=0x7f209c13b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:21:49.067 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.089+0000 7f209affd640 1 -- 172.21.15.47:0/695548842 >> v1:172.21.15.121:6789/0 conn(0x7f209c0a9ac0 legacy=0x7f209c0adfe0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:21:49.067 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.089+0000 7f209affd640 10 monclient: _finish_hunting 0 2024-03-29T11:21:49.067 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.089+0000 7f209affd640 1 monclient: found mon.noname-a 2024-03-29T11:21:49.067 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.089+0000 7f209affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:49.067 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.089+0000 7f209affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-29T11:21:49.068 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.089+0000 7f209affd640 1 -- 172.21.15.47:0/695548842 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f209c0aaf60 con 0x7f209c0af900 2024-03-29T11:21:49.068 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.089+0000 7f209affd640 10 monclient: _finish_auth 0 2024-03-29T11:21:49.068 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.089+0000 7f209affd640 10 monclient: _check_auth_tickets 2024-03-29T11:21:49.068 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.089+0000 7f209affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:49.068 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.089+0000 7f209affd640 1 -- 172.21.15.47:0/695548842 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f2094004d90 con 0x7f209c0af900 2024-03-29T11:21:49.068 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.089+0000 7f209affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:49.068 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.089+0000 7f209affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:49.068 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.089+0000 7f209affd640 10 monclient: dump: 2024-03-29T11:21:49.068 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:21:49.068 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:21:49.068 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:49.068 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:49.068 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:49.068 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:21:49.069 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:21:49.069 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:21:49.069 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:21:49.069 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:21:49.069 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.089+0000 7f20a2367640 5 monclient: authenticate success, global_id 4527 2024-03-29T11:21:49.069 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.089+0000 7f20a2367640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:49.069 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.092+0000 7f209affd640 1 -- 172.21.15.47:0/695548842 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2094004ff0 con 0x7f209c0af900 2024-03-29T11:21:49.070 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.092+0000 7f209affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:49.070 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.092+0000 7f209affd640 1 -- 172.21.15.47:0/695548842 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f2094004d90 con 0x7f209c0af900 2024-03-29T11:21:49.070 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.092+0000 7f209affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:49.070 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.092+0000 7f209affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:49.070 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.092+0000 7f209affd640 10 monclient: dump: 2024-03-29T11:21:49.070 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:21:49.070 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:21:49.070 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:49.070 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:49.070 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:49.070 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:21:49.070 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:21:49.070 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:21:49.070 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:21:49.071 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:21:49.071 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.092+0000 7f20a2367640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:49.071 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.092+0000 7f20a2367640 4 set_mon_vals no callback set 2024-03-29T11:21:49.071 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.093+0000 7f20a2367640 10 monclient: shutdown 2024-03-29T11:21:49.071 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.093+0000 7f20a2367640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:49.071 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.093+0000 7f20a2367640 1 -- 172.21.15.47:0/695548842 >> v1:172.21.15.47:6789/0 conn(0x7f209c0af900 legacy=0x7f209c149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:49.071 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.093+0000 7f20a2367640 1 -- 172.21.15.47:0/695548842 shutdown_connections 2024-03-29T11:21:49.071 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.093+0000 7f20a2367640 1 -- 172.21.15.47:0/695548842 >> 172.21.15.47:0/695548842 conn(0x7f209c0b35e0 msgr2=0x7f209c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:49.071 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.094+0000 7f20a2367640 1 -- 172.21.15.47:0/695548842 shutdown_connections 2024-03-29T11:21:49.073 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.095+0000 7f20a2367640 1 -- 172.21.15.47:0/695548842 wait complete. 2024-03-29T11:21:49.073 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.096+0000 7f20a2367640 10 monclient: build_initial_monmap 2024-03-29T11:21:49.073 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.096+0000 7f20a2367640 10 monclient: monmap: 2024-03-29T11:21:49.073 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:21:49.073 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:49.073 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:21:49.073 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:21:49.073 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:49.073 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:21:49.073 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:21:49.074 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:21:49.074 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:21:49.074 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:21:49.074 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.096+0000 7f20a2367640 1 librados: starting msgr at 2024-03-29T11:21:49.074 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.096+0000 7f20a2367640 1 librados: starting objecter 2024-03-29T11:21:49.074 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.096+0000 7f20a2367640 1 Processor -- start 2024-03-29T11:21:49.074 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.097+0000 7f20a2367640 1 -- start start 2024-03-29T11:21:49.075 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.097+0000 7f20a2367640 1 librados: setting wanted keys 2024-03-29T11:21:49.075 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.097+0000 7f20a2367640 1 librados: calling monclient init 2024-03-29T11:21:49.077 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.097+0000 7f20a2367640 10 monclient: init 2024-03-29T11:21:49.077 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.097+0000 7f20a2367640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:49.077 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.097+0000 7f20a2367640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:21:49.082 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.098+0000 7f20a2367640 10 monclient(hunting): picked mon.noname-c con 0x7f209c0a9ac0 addr v1:172.21.15.47:6790/0 2024-03-29T11:21:49.082 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.100+0000 7f209bfff640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f209c0a9ac0 0x7f209c1d6e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:49361/0 (socket says 172.21.15.47:49361) 2024-03-29T11:21:49.082 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.100+0000 7f20a2367640 10 monclient(hunting): picked mon.noname-a con 0x7f209c0af900 addr v1:172.21.15.47:6789/0 2024-03-29T11:21:49.082 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.100+0000 7f209bfff640 1 -- 172.21.15.47:0/2960605153 learned_addr learned my addr 172.21.15.47:0/2960605153 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:21:49.082 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.100+0000 7f20a2367640 10 monclient(hunting): picked mon.noname-b con 0x7f209c13ee90 addr v1:172.21.15.121:6789/0 2024-03-29T11:21:49.082 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.101+0000 7f20a2367640 1 -- 172.21.15.47:0/2960605153 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f209c135170 con 0x7f209c0af900 2024-03-29T11:21:49.082 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.101+0000 7f20a2367640 1 -- 172.21.15.47:0/2960605153 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f209c140e90 con 0x7f209c13ee90 2024-03-29T11:21:49.082 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.101+0000 7f20a2367640 1 -- 172.21.15.47:0/2960605153 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f209c1d6950 con 0x7f209c0a9ac0 2024-03-29T11:21:49.082 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.101+0000 7f20a2367640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:49.082 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.101+0000 7f20a2367640 10 monclient(hunting): authenticate will time out at 1352.288980s 2024-03-29T11:21:49.082 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.104+0000 7f2098ff9640 10 client.?.objecter ms_handle_connect 0x7f209c0a9ac0 2024-03-29T11:21:49.083 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.104+0000 7f2098ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:49.083 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.104+0000 7f2098ff9640 1 -- 172.21.15.47:0/2960605153 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 330755846 0 0) 0x7f209c1d6950 con 0x7f209c0a9ac0 2024-03-29T11:21:49.083 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.104+0000 7f2098ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:21:49.083 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.104+0000 7f2098ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:49.083 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.104+0000 7f2098ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:49.083 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.104+0000 7f2098ff9640 10 monclient(hunting): my global_id is 4589 2024-03-29T11:21:49.083 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.104+0000 7f2098ff9640 1 -- 172.21.15.47:0/2960605153 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2088004eb0 con 0x7f209c0a9ac0 2024-03-29T11:21:49.083 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.105+0000 7f2098ff9640 10 client.?.objecter ms_handle_connect 0x7f209c13ee90 2024-03-29T11:21:49.083 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.105+0000 7f2098ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:49.085 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.106+0000 7f2098ff9640 10 client.?.objecter ms_handle_connect 0x7f209c0af900 2024-03-29T11:21:49.085 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.106+0000 7f2098ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:49.085 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.106+0000 7f2098ff9640 1 -- 172.21.15.47:0/2960605153 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 609548679 0 0) 0x7f209c135170 con 0x7f209c0af900 2024-03-29T11:21:49.085 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.106+0000 7f2098ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:21:49.086 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.106+0000 7f2098ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:49.086 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.106+0000 7f2098ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:49.086 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.106+0000 7f2098ff9640 10 monclient(hunting): my global_id is 4530 2024-03-29T11:21:49.086 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.106+0000 7f2098ff9640 1 -- 172.21.15.47:0/2960605153 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f209c1d6950 con 0x7f209c0af900 2024-03-29T11:21:49.086 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.107+0000 7f2098ff9640 1 -- 172.21.15.47:0/2960605153 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1514509748 0 0) 0x7f209c140e90 con 0x7f209c13ee90 2024-03-29T11:21:49.086 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.107+0000 7f2098ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:21:49.086 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.107+0000 7f2098ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:49.086 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.107+0000 7f2098ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:49.086 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.107+0000 7f2098ff9640 10 monclient(hunting): my global_id is 4516 2024-03-29T11:21:49.086 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.107+0000 7f2098ff9640 1 -- 172.21.15.47:0/2960605153 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f209c135170 con 0x7f209c13ee90 2024-03-29T11:21:49.086 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.107+0000 7f2098ff9640 1 -- 172.21.15.47:0/2960605153 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1091052986 0 0) 0x7f2088004eb0 con 0x7f209c0a9ac0 2024-03-29T11:21:49.086 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.107+0000 7f2098ff9640 1 -- 172.21.15.47:0/2960605153 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f209c140e90 con 0x7f209c0a9ac0 2024-03-29T11:21:49.086 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.107+0000 7f2098ff9640 1 -- 172.21.15.47:0/2960605153 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f208c0017f0 con 0x7f209c0a9ac0 2024-03-29T11:21:49.086 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.107+0000 7f2098ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:49.087 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.107+0000 7f2098ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:49.087 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.107+0000 7f2098ff9640 10 monclient(hunting): dump: 2024-03-29T11:21:49.087 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:21:49.087 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:21:49.087 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:49.087 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:49.087 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:49.087 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:21:49.087 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:21:49.087 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:21:49.087 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:21:49.087 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:21:49.087 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.107+0000 7f2098ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:49.088 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.108+0000 7f2098ff9640 1 -- 172.21.15.47:0/2960605153 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 824015814 0 0) 0x7f209c1d6950 con 0x7f209c0af900 2024-03-29T11:21:49.088 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.108+0000 7f2098ff9640 1 -- 172.21.15.47:0/2960605153 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2088004eb0 con 0x7f209c0af900 2024-03-29T11:21:49.088 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.108+0000 7f2098ff9640 1 -- 172.21.15.47:0/2960605153 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3260570187 0 0) 0x7f209c135170 con 0x7f209c13ee90 2024-03-29T11:21:49.089 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.108+0000 7f2098ff9640 1 -- 172.21.15.47:0/2960605153 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f209c1d6950 con 0x7f209c13ee90 2024-03-29T11:21:49.089 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.108+0000 7f2098ff9640 1 -- 172.21.15.47:0/2960605153 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3641439596 0 0) 0x7f209c140e90 con 0x7f209c0a9ac0 2024-03-29T11:21:49.089 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.108+0000 7f2098ff9640 1 -- 172.21.15.47:0/2960605153 >> v1:172.21.15.121:6789/0 conn(0x7f209c13ee90 legacy=0x7f209c134a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:49.089 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.108+0000 7f2098ff9640 1 -- 172.21.15.47:0/2960605153 >> v1:172.21.15.47:6789/0 conn(0x7f209c0af900 legacy=0x7f209c134350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:49.089 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.108+0000 7f2098ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:21:49.089 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.108+0000 7f2098ff9640 1 monclient: found mon.c 2024-03-29T11:21:49.089 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.108+0000 7f2098ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:49.089 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.108+0000 7f2098ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:21:49.089 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.108+0000 7f2098ff9640 1 -- 172.21.15.47:0/2960605153 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f209c1d7580 con 0x7f209c0a9ac0 2024-03-29T11:21:49.089 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.108+0000 7f2098ff9640 10 monclient: _finish_auth 0 2024-03-29T11:21:49.089 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.108+0000 7f2098ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:21:49.089 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.108+0000 7f2098ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:49.089 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.111+0000 7f20a2367640 5 monclient: authenticate success, global_id 4589 2024-03-29T11:21:49.089 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.111+0000 7f20a2367640 10 monclient: _renew_subs 2024-03-29T11:21:49.089 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.111+0000 7f20a2367640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:21:49.090 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.111+0000 7f20a2367640 1 -- 172.21.15.47:0/2960605153 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f209c13f830 con 0x7f209c0a9ac0 2024-03-29T11:21:49.091 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.112+0000 7f2098ff9640 1 -- 172.21.15.47:0/2960605153 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f208c001950 con 0x7f209c0a9ac0 2024-03-29T11:21:49.091 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.112+0000 7f2098ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:49.091 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.112+0000 7f209a7fc640 4 set_mon_vals no callback set 2024-03-29T11:21:49.092 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.112+0000 7f20a2367640 10 client.4589.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:49.092 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.113+0000 7f2098ff9640 1 -- 172.21.15.47:0/2960605153 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f208c001e00 con 0x7f209c0a9ac0 2024-03-29T11:21:49.092 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.113+0000 7f2098ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:49.092 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.113+0000 7f2098ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:49.092 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.113+0000 7f2098ff9640 10 monclient: dump: 2024-03-29T11:21:49.092 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:21:49.092 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:21:49.092 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:49.092 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:49.092 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:49.092 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:21:49.092 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:21:49.092 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:21:49.092 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:21:49.092 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:21:49.093 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.114+0000 7f20a2367640 10 monclient: _renew_subs 2024-03-29T11:21:49.093 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.114+0000 7f20a2367640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:21:49.093 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.114+0000 7f20a2367640 1 -- 172.21.15.47:0/2960605153 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f209c141490 con 0x7f209c0a9ac0 2024-03-29T11:21:49.093 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.114+0000 7f20a2367640 1 librados: init done 2024-03-29T11:21:49.093 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.114+0000 7f2098ff9640 1 -- 172.21.15.47:0/2960605153 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7f208c01a9d0 con 0x7f209c0a9ac0 2024-03-29T11:21:49.093 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.114+0000 7f2098ff9640 1 -- 172.21.15.47:0/2960605153 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7f208c085a00 con 0x7f209c0a9ac0 2024-03-29T11:21:49.093 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.114+0000 7f2098ff9640 10 client.4589.objecter ms_dispatch 0x7f209c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:49.093 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.114+0000 7f2098ff9640 3 client.4589.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:49.093 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.114+0000 7f2098ff9640 3 client.4589.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:49.093 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.114+0000 7f2098ff9640 20 client.4589.objecter dump_active .. 0 homeless 2024-03-29T11:21:49.094 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.115+0000 7f20a2367640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:49.094 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.116+0000 7f20a2367640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:49.094 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.116+0000 7f20a2367640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:21:49.094 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.116+0000 7f20a2367640 1 -- 172.21.15.47:0/2960605153 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f209c13fbd0 con 0x7f209c0a9ac0 2024-03-29T11:21:49.094 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.117+0000 7f209b7fe640 10 monclient: get_auth_request con 0x7f208806dee0 auth_method 0 2024-03-29T11:21:49.094 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.117+0000 7f2098ff9640 10 client.4589.objecter ms_handle_connect 0x7f208806dee0 2024-03-29T11:21:49.099 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.123+0000 7f2098ff9640 1 -- 172.21.15.47:0/2960605153 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f208c0502b0 con 0x7f209c0a9ac0 2024-03-29T11:21:49.100 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.123+0000 7f2098ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:49.100 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.123+0000 7f2098ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:49.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.443+0000 7f20a2367640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:49.420 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.443+0000 7f20a2367640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:49.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.443+0000 7f20a2367640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:21:49.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.443+0000 7f20a2367640 1 -- 172.21.15.47:0/2960605153 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f209c1d6ae0 con 0x7f209c0a9ac0 2024-03-29T11:21:49.423 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.444+0000 7f2098ff9640 1 -- 172.21.15.47:0/2960605153 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7f208c0521e0 con 0x7f209c0a9ac0 2024-03-29T11:21:49.423 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.444+0000 7f2098ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:49.423 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.444+0000 7f2098ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:49.423 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:21:49.424 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:49.424 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.447+0000 7f20a2367640 10 librados: watch_flush enter 2024-03-29T11:21:49.425 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.448+0000 7f20a2367640 10 librados: watch_flush exit 2024-03-29T11:21:49.425 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.448+0000 7f20a2367640 20 client.4589.objecter shutdown clearing up homeless session... 2024-03-29T11:21:49.425 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.448+0000 7f20a2367640 10 client.4589.objecter successfully canceled tick 2024-03-29T11:21:49.425 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.448+0000 7f20a2367640 1 -- 172.21.15.47:0/2960605153 >> v1:172.21.15.121:6816/1964117370 conn(0x7f208806dee0 legacy=0x7f20880703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:49.425 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.448+0000 7f20a2367640 10 monclient: shutdown 2024-03-29T11:21:49.425 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.448+0000 7f20a2367640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:49.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.448+0000 7f20a2367640 1 -- 172.21.15.47:0/2960605153 >> v1:172.21.15.47:6790/0 conn(0x7f209c0a9ac0 legacy=0x7f209c1d6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:49.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.448+0000 7f20a2367640 1 -- 172.21.15.47:0/2960605153 shutdown_connections 2024-03-29T11:21:49.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.448+0000 7f20a2367640 1 -- 172.21.15.47:0/2960605153 >> 172.21.15.47:0/2960605153 conn(0x7f209c0b35e0 msgr2=0x7f209c0ad0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:49.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.451+0000 7f20a2367640 1 -- 172.21.15.47:0/2960605153 shutdown_connections 2024-03-29T11:21:49.428 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.451+0000 7f20a2367640 1 -- 172.21.15.47:0/2960605153 wait complete. 2024-03-29T11:21:49.431 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:49.451+0000 7f20a2367640 1 librados: shutdown 2024-03-29T11:21:49.447 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:56.254 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:56.399 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.422+0000 7f70608b5640 10 monclient: get_monmap_and_config 2024-03-29T11:21:56.399 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.422+0000 7f70608b5640 10 monclient: build_initial_monmap 2024-03-29T11:21:56.399 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.422+0000 7f70608b5640 10 monclient: monmap: 2024-03-29T11:21:56.399 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:21:56.399 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:21:56.399 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:21:56.423406+0000 2024-03-29T11:21:56.400 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:21:56.423406+0000 2024-03-29T11:21:56.400 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:56.400 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:21:56.400 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:21:56.400 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:21:56.400 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:21:56.400 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:21:56.401 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.423+0000 7f70608b5640 1 Processor -- start 2024-03-29T11:21:56.401 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.423+0000 7f70608b5640 1 -- start start 2024-03-29T11:21:56.401 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.423+0000 7f70608b5640 10 monclient: init 2024-03-29T11:21:56.401 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.424+0000 7f70608b5640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:56.401 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.424+0000 7f70608b5640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:21:56.402 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.425+0000 7f70608b5640 10 monclient(hunting): picked mon.noname-a con 0x7f705c06d9d0 addr v1:172.21.15.47:6789/0 2024-03-29T11:21:56.402 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.425+0000 7f70608b5640 10 monclient(hunting): picked mon.noname-b con 0x7f705c13ef30 addr v1:172.21.15.47:6790/0 2024-03-29T11:21:56.402 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.425+0000 7f7053fff640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f705c13ef30 0x7f705c0af1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:52254/0 (socket says 172.21.15.47:52254) 2024-03-29T11:21:56.402 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.425+0000 7f7053fff640 1 -- 172.21.15.47:0/1338734559 learned_addr learned my addr 172.21.15.47:0/1338734559 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:21:56.403 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.426+0000 7f70608b5640 10 monclient(hunting): picked mon.noname-c con 0x7f705c0b07d0 addr v1:172.21.15.121:6789/0 2024-03-29T11:21:56.404 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.426+0000 7f70608b5640 1 -- 172.21.15.47:0/1338734559 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f705c0aed10 con 0x7f705c06d9d0 2024-03-29T11:21:56.404 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.426+0000 7f70608b5640 1 -- 172.21.15.47:0/1338734559 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f705c0b0c40 con 0x7f705c0b07d0 2024-03-29T11:21:56.404 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.426+0000 7f70608b5640 1 -- 172.21.15.47:0/1338734559 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f705c0ab1b0 con 0x7f705c13ef30 2024-03-29T11:21:56.404 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.426+0000 7f70608b5640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:56.404 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.426+0000 7f70608b5640 10 monclient(hunting): authenticate will time out at 1359.614699s 2024-03-29T11:21:56.404 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.427+0000 7f7059d9b640 1 -- 172.21.15.47:0/1338734559 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2340572758 0 0) 0x7f705c0ab1b0 con 0x7f705c13ef30 2024-03-29T11:21:56.404 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.427+0000 7f7059d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:21:56.404 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.427+0000 7f7059d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:56.404 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.427+0000 7f7059d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:56.404 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.427+0000 7f7059d9b640 10 monclient(hunting): my global_id is 4592 2024-03-29T11:21:56.404 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.427+0000 7f7059d9b640 1 -- 172.21.15.47:0/1338734559 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7048005730 con 0x7f705c13ef30 2024-03-29T11:21:56.404 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.427+0000 7f7059d9b640 1 -- 172.21.15.47:0/1338734559 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2746585445 0 0) 0x7f7048005730 con 0x7f705c13ef30 2024-03-29T11:21:56.404 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.427+0000 7f7059d9b640 1 -- 172.21.15.47:0/1338734559 >> v1:172.21.15.121:6789/0 conn(0x7f705c0b07d0 legacy=0x7f705c14ab80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:21:56.408 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.431+0000 7f7059d9b640 1 -- 172.21.15.47:0/1338734559 >> v1:172.21.15.47:6789/0 conn(0x7f705c06d9d0 legacy=0x7f705c13e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:56.409 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.431+0000 7f7059d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:21:56.409 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.431+0000 7f7059d9b640 1 monclient: found mon.noname-b 2024-03-29T11:21:56.409 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.431+0000 7f7059d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:56.409 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.431+0000 7f7059d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-29T11:21:56.409 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.431+0000 7f7059d9b640 1 -- 172.21.15.47:0/1338734559 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f705c0ab490 con 0x7f705c13ef30 2024-03-29T11:21:56.409 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.431+0000 7f7059d9b640 10 monclient: _finish_auth 0 2024-03-29T11:21:56.409 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.431+0000 7f7059d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:21:56.409 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.431+0000 7f7059d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:56.409 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.431+0000 7f70608b5640 5 monclient: authenticate success, global_id 4592 2024-03-29T11:21:56.409 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.431+0000 7f7059d9b640 1 -- 172.21.15.47:0/1338734559 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f7044001690 con 0x7f705c13ef30 2024-03-29T11:21:56.409 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.431+0000 7f70608b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:56.409 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.431+0000 7f7059d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:56.409 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.431+0000 7f7059d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:56.409 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.431+0000 7f7059d9b640 10 monclient: dump: 2024-03-29T11:21:56.410 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:21:56.410 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:21:56.410 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:56.410 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:56.410 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:56.410 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:21:56.410 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:21:56.410 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:21:56.410 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:21:56.410 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:21:56.410 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.431+0000 7f70608b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:56.410 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.432+0000 7f7059d9b640 1 -- 172.21.15.47:0/1338734559 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7044001860 con 0x7f705c13ef30 2024-03-29T11:21:56.410 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.432+0000 7f7059d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:56.410 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.432+0000 7f7059d9b640 1 -- 172.21.15.47:0/1338734559 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f7044001cb0 con 0x7f705c13ef30 2024-03-29T11:21:56.411 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.432+0000 7f7059d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:56.411 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.432+0000 7f7059d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:56.411 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.432+0000 7f7059d9b640 10 monclient: dump: 2024-03-29T11:21:56.411 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:21:56.411 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:21:56.411 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:56.411 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:56.411 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:56.411 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:21:56.411 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:21:56.411 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:21:56.411 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:21:56.411 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:21:56.411 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.432+0000 7f70608b5640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:56.412 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.432+0000 7f70608b5640 4 set_mon_vals no callback set 2024-03-29T11:21:56.413 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.433+0000 7f70608b5640 10 monclient: shutdown 2024-03-29T11:21:56.414 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.433+0000 7f70608b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:56.414 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.433+0000 7f70608b5640 1 -- 172.21.15.47:0/1338734559 >> v1:172.21.15.47:6790/0 conn(0x7f705c13ef30 legacy=0x7f705c0af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:56.414 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.433+0000 7f70608b5640 1 -- 172.21.15.47:0/1338734559 shutdown_connections 2024-03-29T11:21:56.414 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.433+0000 7f70608b5640 1 -- 172.21.15.47:0/1338734559 >> 172.21.15.47:0/1338734559 conn(0x7f705c06b660 msgr2=0x7f705c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:56.414 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.433+0000 7f70608b5640 1 -- 172.21.15.47:0/1338734559 shutdown_connections 2024-03-29T11:21:56.414 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.433+0000 7f70608b5640 1 -- 172.21.15.47:0/1338734559 wait complete. 2024-03-29T11:21:56.414 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.434+0000 7f70608b5640 10 monclient: build_initial_monmap 2024-03-29T11:21:56.414 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.435+0000 7f70608b5640 10 monclient: monmap: 2024-03-29T11:21:56.414 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:21:56.414 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:56.414 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:21:56.414 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:21:56.414 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:56.414 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:21:56.415 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:21:56.415 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:21:56.415 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:21:56.415 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:21:56.415 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.435+0000 7f70608b5640 1 librados: starting msgr at 2024-03-29T11:21:56.415 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.435+0000 7f70608b5640 1 librados: starting objecter 2024-03-29T11:21:56.415 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.435+0000 7f70608b5640 1 Processor -- start 2024-03-29T11:21:56.415 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.436+0000 7f70608b5640 1 -- start start 2024-03-29T11:21:56.415 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.436+0000 7f70608b5640 1 librados: setting wanted keys 2024-03-29T11:21:56.415 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.436+0000 7f70608b5640 1 librados: calling monclient init 2024-03-29T11:21:56.415 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.436+0000 7f70608b5640 10 monclient: init 2024-03-29T11:21:56.415 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.437+0000 7f70608b5640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:56.415 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.437+0000 7f70608b5640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:21:56.417 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.438+0000 7f70608b5640 10 monclient(hunting): picked mon.noname-a con 0x7f705c06d9d0 addr v1:172.21.15.47:6789/0 2024-03-29T11:21:56.417 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.438+0000 7f70608b5640 10 monclient(hunting): picked mon.noname-b con 0x7f705c0b07d0 addr v1:172.21.15.121:6789/0 2024-03-29T11:21:56.417 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.438+0000 7f705ad9d640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f705c13ef30 0x7f705c1de5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:53624/0 (socket says 172.21.15.47:53624) 2024-03-29T11:21:56.417 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.438+0000 7f705ad9d640 1 -- 172.21.15.47:0/2214518216 learned_addr learned my addr 172.21.15.47:0/2214518216 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:21:56.417 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.438+0000 7f70608b5640 10 monclient(hunting): picked mon.noname-c con 0x7f705c13ef30 addr v1:172.21.15.47:6790/0 2024-03-29T11:21:56.417 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.438+0000 7f70608b5640 1 -- 172.21.15.47:0/2214518216 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f705c0b3fa0 con 0x7f705c06d9d0 2024-03-29T11:21:56.417 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.438+0000 7f70608b5640 1 -- 172.21.15.47:0/2214518216 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f705c06f010 con 0x7f705c0b07d0 2024-03-29T11:21:56.417 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.438+0000 7f70608b5640 1 -- 172.21.15.47:0/2214518216 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f705c0b3080 con 0x7f705c13ef30 2024-03-29T11:21:56.418 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.438+0000 7f70608b5640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:56.418 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.438+0000 7f70608b5640 10 monclient(hunting): authenticate will time out at 1359.626636s 2024-03-29T11:21:56.418 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.439+0000 7f7052ffd640 10 client.?.objecter ms_handle_connect 0x7f705c13ef30 2024-03-29T11:21:56.418 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.439+0000 7f7052ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:56.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.441+0000 7f7052ffd640 1 -- 172.21.15.47:0/2214518216 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1594200665 0 0) 0x7f705c0b3080 con 0x7f705c13ef30 2024-03-29T11:21:56.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.441+0000 7f7052ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:56.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.441+0000 7f7052ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:56.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.441+0000 7f7052ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:56.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.441+0000 7f7052ffd640 10 monclient(hunting): my global_id is 4595 2024-03-29T11:21:56.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.441+0000 7f7052ffd640 1 -- 172.21.15.47:0/2214518216 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f703c004eb0 con 0x7f705c13ef30 2024-03-29T11:21:56.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.442+0000 7f7052ffd640 1 -- 172.21.15.47:0/2214518216 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 469016256 0 0) 0x7f703c004eb0 con 0x7f705c13ef30 2024-03-29T11:21:56.423 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.442+0000 7f7052ffd640 1 -- 172.21.15.47:0/2214518216 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f705c0b3080 con 0x7f705c13ef30 2024-03-29T11:21:56.423 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.442+0000 7f7052ffd640 1 -- 172.21.15.47:0/2214518216 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f7054001b70 con 0x7f705c13ef30 2024-03-29T11:21:56.423 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.442+0000 7f7052ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:56.423 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.442+0000 7f7052ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:56.423 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.442+0000 7f7052ffd640 10 monclient(hunting): dump: 2024-03-29T11:21:56.423 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:21:56.423 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:21:56.423 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:56.423 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:56.423 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:56.423 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:21:56.423 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:21:56.423 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:21:56.423 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:21:56.424 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:21:56.424 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.442+0000 7f7052ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:56.424 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.442+0000 7f7052ffd640 1 -- 172.21.15.47:0/2214518216 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1742621873 0 0) 0x7f705c0b3080 con 0x7f705c13ef30 2024-03-29T11:21:56.424 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.442+0000 7f7052ffd640 1 -- 172.21.15.47:0/2214518216 >> v1:172.21.15.121:6789/0 conn(0x7f705c0b07d0 legacy=0x7f705c1d62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:56.424 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.442+0000 7f7052ffd640 1 -- 172.21.15.47:0/2214518216 >> v1:172.21.15.47:6789/0 conn(0x7f705c06d9d0 legacy=0x7f705c1326f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:56.424 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.443+0000 7f7052ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:21:56.424 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.443+0000 7f7052ffd640 1 monclient: found mon.c 2024-03-29T11:21:56.424 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.443+0000 7f7052ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:56.424 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.443+0000 7f7052ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:21:56.424 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.443+0000 7f7052ffd640 1 -- 172.21.15.47:0/2214518216 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f705c132e00 con 0x7f705c13ef30 2024-03-29T11:21:56.424 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.443+0000 7f7052ffd640 10 monclient: _finish_auth 0 2024-03-29T11:21:56.424 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.443+0000 7f7052ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:21:56.424 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.443+0000 7f7052ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:56.424 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.443+0000 7f7052ffd640 1 -- 172.21.15.47:0/2214518216 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7054001cd0 con 0x7f705c13ef30 2024-03-29T11:21:56.424 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.443+0000 7f7052ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:56.425 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.443+0000 7f7052ffd640 1 -- 172.21.15.47:0/2214518216 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f7054004e20 con 0x7f705c13ef30 2024-03-29T11:21:56.425 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.443+0000 7f7052ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:56.425 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.443+0000 7f7052ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:56.425 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.443+0000 7f7052ffd640 10 monclient: dump: 2024-03-29T11:21:56.425 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:21:56.425 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:21:56.425 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:56.425 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:21:56.425 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:56.425 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:21:56.425 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:21:56.426 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:21:56.426 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:21:56.426 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:21:56.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.443+0000 7f705959a640 4 set_mon_vals no callback set 2024-03-29T11:21:56.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.443+0000 7f70608b5640 5 monclient: authenticate success, global_id 4595 2024-03-29T11:21:56.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.443+0000 7f70608b5640 10 monclient: _renew_subs 2024-03-29T11:21:56.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.443+0000 7f70608b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:21:56.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.443+0000 7f70608b5640 1 -- 172.21.15.47:0/2214518216 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f705c133050 con 0x7f705c13ef30 2024-03-29T11:21:56.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.444+0000 7f70608b5640 10 client.4595.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:56.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.444+0000 7f70608b5640 10 monclient: _renew_subs 2024-03-29T11:21:56.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.444+0000 7f70608b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:21:56.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.444+0000 7f70608b5640 1 -- 172.21.15.47:0/2214518216 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f705c138330 con 0x7f705c13ef30 2024-03-29T11:21:56.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.444+0000 7f70608b5640 1 librados: init done 2024-03-29T11:21:56.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.446+0000 7f7050ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:56.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.446+0000 7f7050ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:56.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.446+0000 7f7050ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:21:56.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.446+0000 7f7050ff9640 1 -- 172.21.15.47:0/2214518216 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f705c06ee10 con 0x7f705c13ef30 2024-03-29T11:21:56.428 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.447+0000 7f7052ffd640 1 -- 172.21.15.47:0/2214518216 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7f705401adf0 con 0x7f705c13ef30 2024-03-29T11:21:56.429 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.448+0000 7f705a59c640 10 monclient: get_auth_request con 0x7f703c06de50 auth_method 0 2024-03-29T11:21:56.429 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.448+0000 7f7052ffd640 1 -- 172.21.15.47:0/2214518216 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7f7054084e30 con 0x7f705c13ef30 2024-03-29T11:21:56.429 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.449+0000 7f7052ffd640 10 client.4595.objecter ms_dispatch 0x7f705c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:56.429 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.449+0000 7f7052ffd640 3 client.4595.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:56.429 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.449+0000 7f7052ffd640 3 client.4595.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:56.429 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.449+0000 7f7052ffd640 20 client.4595.objecter dump_active .. 0 homeless 2024-03-29T11:21:56.429 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.449+0000 7f7052ffd640 10 client.4595.objecter ms_handle_connect 0x7f703c06de50 2024-03-29T11:21:56.432 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.455+0000 7f7052ffd640 1 -- 172.21.15.47:0/2214518216 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7054050540 con 0x7f705c13ef30 2024-03-29T11:21:56.432 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.455+0000 7f7052ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:56.432 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.455+0000 7f7052ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:56.773 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.797+0000 7f7050ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:56.774 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.797+0000 7f7050ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:56.774 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.797+0000 7f7050ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:21:56.774 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.797+0000 7f7050ff9640 1 -- 172.21.15.47:0/2214518216 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f705c0b4130 con 0x7f705c13ef30 2024-03-29T11:21:56.774 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.798+0000 7f7052ffd640 1 -- 172.21.15.47:0/2214518216 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7f7054052640 con 0x7f705c13ef30 2024-03-29T11:21:56.774 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.798+0000 7f7052ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:56.775 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.798+0000 7f7052ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:56.775 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:21:56.775 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:56.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.801+0000 7f7050ff9640 10 librados: watch_flush enter 2024-03-29T11:21:56.779 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.802+0000 7f7050ff9640 10 librados: watch_flush exit 2024-03-29T11:21:56.779 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.802+0000 7f7050ff9640 20 client.4595.objecter shutdown clearing up homeless session... 2024-03-29T11:21:56.779 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.802+0000 7f7050ff9640 10 client.4595.objecter successfully canceled tick 2024-03-29T11:21:56.779 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.802+0000 7f7050ff9640 1 -- 172.21.15.47:0/2214518216 >> v1:172.21.15.121:6816/1964117370 conn(0x7f703c06de50 legacy=0x7f703c070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:56.779 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.802+0000 7f7050ff9640 10 monclient: shutdown 2024-03-29T11:21:56.779 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.802+0000 7f7050ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:56.779 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.802+0000 7f7050ff9640 1 -- 172.21.15.47:0/2214518216 >> v1:172.21.15.47:6790/0 conn(0x7f705c13ef30 legacy=0x7f705c1de5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:56.779 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.802+0000 7f7050ff9640 1 -- 172.21.15.47:0/2214518216 shutdown_connections 2024-03-29T11:21:56.779 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.802+0000 7f7050ff9640 1 -- 172.21.15.47:0/2214518216 >> 172.21.15.47:0/2214518216 conn(0x7f705c06b660 msgr2=0x7f705c13e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:56.779 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.802+0000 7f7050ff9640 1 -- 172.21.15.47:0/2214518216 shutdown_connections 2024-03-29T11:21:56.779 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.802+0000 7f7050ff9640 1 -- 172.21.15.47:0/2214518216 wait complete. 2024-03-29T11:21:56.779 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:21:56.802+0000 7f7050ff9640 1 librados: shutdown 2024-03-29T11:21:56.799 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:03.606 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:03.761 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.785+0000 7f39e396b640 10 monclient: get_monmap_and_config 2024-03-29T11:22:03.761 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.785+0000 7f39e396b640 10 monclient: build_initial_monmap 2024-03-29T11:22:03.762 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.785+0000 7f39e396b640 10 monclient: monmap: 2024-03-29T11:22:03.762 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:22:03.762 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:22:03.762 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:22:03.786040+0000 2024-03-29T11:22:03.762 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:22:03.786040+0000 2024-03-29T11:22:03.762 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:03.762 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:22:03.762 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:22:03.762 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:22:03.762 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:22:03.763 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:22:03.763 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.786+0000 7f39e396b640 1 Processor -- start 2024-03-29T11:22:03.763 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.786+0000 7f39e396b640 1 -- start start 2024-03-29T11:22:03.764 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.786+0000 7f39e396b640 10 monclient: init 2024-03-29T11:22:03.764 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.787+0000 7f39e396b640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:03.764 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.787+0000 7f39e396b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:22:03.764 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.787+0000 7f39e396b640 10 monclient(hunting): picked mon.noname-b con 0x7f39dc1457f0 addr v1:172.21.15.47:6790/0 2024-03-29T11:22:03.764 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.787+0000 7f39e396b640 10 monclient(hunting): picked mon.noname-a con 0x7f39dc13f650 addr v1:172.21.15.47:6789/0 2024-03-29T11:22:03.764 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.787+0000 7f39e396b640 10 monclient(hunting): picked mon.noname-c con 0x7f39dc0a9d70 addr v1:172.21.15.121:6789/0 2024-03-29T11:22:03.764 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.787+0000 7f39e396b640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39dc0aaef0 con 0x7f39dc13f650 2024-03-29T11:22:03.764 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.787+0000 7f39e396b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39dc145bd0 con 0x7f39dc0a9d70 2024-03-29T11:22:03.764 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.787+0000 7f39e396b640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39dc145eb0 con 0x7f39dc1457f0 2024-03-29T11:22:03.764 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.787+0000 7f39e396b640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:03.764 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.787+0000 7f39e396b640 10 monclient(hunting): authenticate will time out at 1366.975273s 2024-03-29T11:22:03.765 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.787+0000 7f39e1f08640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f39dc0a9d70 0x7f39dc14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.47:44970/0 (socket says 172.21.15.47:44970) 2024-03-29T11:22:03.765 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.787+0000 7f39e1f08640 1 -- 172.21.15.47:0/2329053604 learned_addr learned my addr 172.21.15.47:0/2329053604 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:22:03.765 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.788+0000 7f39d3fff640 1 -- 172.21.15.47:0/2329053604 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3779461231 0 0) 0x7f39dc145bd0 con 0x7f39dc0a9d70 2024-03-29T11:22:03.765 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.788+0000 7f39d3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:03.766 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.788+0000 7f39d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:03.766 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.788+0000 7f39d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:03.766 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.788+0000 7f39d3fff640 10 monclient(hunting): my global_id is 4519 2024-03-29T11:22:03.766 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.788+0000 7f39d3fff640 1 -- 172.21.15.47:0/2329053604 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39c8003520 con 0x7f39dc0a9d70 2024-03-29T11:22:03.766 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.788+0000 7f39d3fff640 1 -- 172.21.15.47:0/2329053604 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3837270787 0 0) 0x7f39dc145eb0 con 0x7f39dc1457f0 2024-03-29T11:22:03.766 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.788+0000 7f39d3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:03.766 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.788+0000 7f39d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:03.766 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.788+0000 7f39d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:03.766 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.788+0000 7f39d3fff640 10 monclient(hunting): my global_id is 4598 2024-03-29T11:22:03.766 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.788+0000 7f39d3fff640 1 -- 172.21.15.47:0/2329053604 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39dc145bd0 con 0x7f39dc1457f0 2024-03-29T11:22:03.766 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.788+0000 7f39d3fff640 1 -- 172.21.15.47:0/2329053604 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1003347664 0 0) 0x7f39dc145bd0 con 0x7f39dc1457f0 2024-03-29T11:22:03.766 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.788+0000 7f39d3fff640 1 -- 172.21.15.47:0/2329053604 >> v1:172.21.15.121:6789/0 conn(0x7f39dc0a9d70 legacy=0x7f39dc14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:03.767 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.788+0000 7f39d3fff640 1 -- 172.21.15.47:0/2329053604 >> v1:172.21.15.47:6789/0 conn(0x7f39dc13f650 legacy=0x7f39dc0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:03.767 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.788+0000 7f39d3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:22:03.767 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.788+0000 7f39d3fff640 1 monclient: found mon.noname-b 2024-03-29T11:22:03.767 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.788+0000 7f39d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:03.767 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.788+0000 7f39d3fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-29T11:22:03.767 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.788+0000 7f39d3fff640 1 -- 172.21.15.47:0/2329053604 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39dc146340 con 0x7f39dc1457f0 2024-03-29T11:22:03.767 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.788+0000 7f39d3fff640 10 monclient: _finish_auth 0 2024-03-29T11:22:03.767 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.788+0000 7f39d3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:22:03.767 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.788+0000 7f39d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:03.767 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.788+0000 7f39d3fff640 1 -- 172.21.15.47:0/2329053604 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f39d80015f0 con 0x7f39dc1457f0 2024-03-29T11:22:03.767 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.788+0000 7f39d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:03.767 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.788+0000 7f39d3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:03.767 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.788+0000 7f39d3fff640 10 monclient: dump: 2024-03-29T11:22:03.767 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:22:03.768 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:22:03.768 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:03.768 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:03.768 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:03.768 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:22:03.768 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:22:03.768 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:22:03.768 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:22:03.768 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:22:03.768 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.788+0000 7f39e396b640 5 monclient: authenticate success, global_id 4598 2024-03-29T11:22:03.768 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.788+0000 7f39e396b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:03.768 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.789+0000 7f39d3fff640 1 -- 172.21.15.47:0/2329053604 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f39d8001750 con 0x7f39dc1457f0 2024-03-29T11:22:03.768 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.789+0000 7f39d3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:03.768 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.789+0000 7f39d3fff640 1 -- 172.21.15.47:0/2329053604 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f39d8001b20 con 0x7f39dc1457f0 2024-03-29T11:22:03.769 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.789+0000 7f39d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:03.769 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.789+0000 7f39d3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:03.769 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.789+0000 7f39d3fff640 10 monclient: dump: 2024-03-29T11:22:03.769 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:22:03.769 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:22:03.769 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:03.769 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:03.769 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:03.769 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:22:03.769 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:22:03.769 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:22:03.769 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:22:03.769 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:22:03.770 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.789+0000 7f39e396b640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:03.770 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.789+0000 7f39e396b640 4 set_mon_vals no callback set 2024-03-29T11:22:03.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.790+0000 7f39e396b640 10 monclient: shutdown 2024-03-29T11:22:03.778 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.790+0000 7f39e396b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:03.778 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.790+0000 7f39e396b640 1 -- 172.21.15.47:0/2329053604 >> v1:172.21.15.47:6790/0 conn(0x7f39dc1457f0 legacy=0x7f39dc13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:03.778 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.790+0000 7f39e396b640 1 -- 172.21.15.47:0/2329053604 shutdown_connections 2024-03-29T11:22:03.778 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.790+0000 7f39e396b640 1 -- 172.21.15.47:0/2329053604 >> 172.21.15.47:0/2329053604 conn(0x7f39dc066a00 msgr2=0x7f39dc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:03.778 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.790+0000 7f39e396b640 1 -- 172.21.15.47:0/2329053604 shutdown_connections 2024-03-29T11:22:03.778 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.790+0000 7f39e396b640 1 -- 172.21.15.47:0/2329053604 wait complete. 2024-03-29T11:22:03.778 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.791+0000 7f39e396b640 10 monclient: build_initial_monmap 2024-03-29T11:22:03.778 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.791+0000 7f39e396b640 10 monclient: monmap: 2024-03-29T11:22:03.778 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:22:03.778 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:03.778 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:22:03.779 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:22:03.779 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:03.779 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:22:03.779 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:22:03.779 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:22:03.779 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:22:03.779 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:22:03.779 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.791+0000 7f39e396b640 1 librados: starting msgr at 2024-03-29T11:22:03.779 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.791+0000 7f39e396b640 1 librados: starting objecter 2024-03-29T11:22:03.779 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.791+0000 7f39e396b640 1 Processor -- start 2024-03-29T11:22:03.779 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.792+0000 7f39e396b640 1 -- start start 2024-03-29T11:22:03.779 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.792+0000 7f39e396b640 1 librados: setting wanted keys 2024-03-29T11:22:03.779 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.792+0000 7f39e396b640 1 librados: calling monclient init 2024-03-29T11:22:03.779 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.792+0000 7f39e396b640 10 monclient: init 2024-03-29T11:22:03.779 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.792+0000 7f39e396b640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:03.780 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.792+0000 7f39e396b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:22:03.780 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.792+0000 7f39e396b640 10 monclient(hunting): picked mon.noname-a con 0x7f39dc0a9d70 addr v1:172.21.15.47:6789/0 2024-03-29T11:22:03.780 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.793+0000 7f39e1707640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f39dc0a9d70 0x7f39dc141940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:36480/0 (socket says 172.21.15.47:36480) 2024-03-29T11:22:03.780 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.793+0000 7f39e396b640 10 monclient(hunting): picked mon.noname-c con 0x7f39dc13f650 addr v1:172.21.15.47:6790/0 2024-03-29T11:22:03.780 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.793+0000 7f39e396b640 10 monclient(hunting): picked mon.noname-b con 0x7f39dc1457f0 addr v1:172.21.15.121:6789/0 2024-03-29T11:22:03.780 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.793+0000 7f39e396b640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39dc1cbd10 con 0x7f39dc0a9d70 2024-03-29T11:22:03.780 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.793+0000 7f39e396b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39dc1cbff0 con 0x7f39dc1457f0 2024-03-29T11:22:03.780 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.793+0000 7f39e1707640 1 -- 172.21.15.47:0/748854794 learned_addr learned my addr 172.21.15.47:0/748854794 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:22:03.780 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.793+0000 7f39e396b640 1 -- 172.21.15.47:0/748854794 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39dc1404c0 con 0x7f39dc13f650 2024-03-29T11:22:03.780 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.793+0000 7f39e396b640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:03.780 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.793+0000 7f39e396b640 10 monclient(hunting): authenticate will time out at 1366.981317s 2024-03-29T11:22:03.780 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.793+0000 7f39d1ffb640 10 client.?.objecter ms_handle_connect 0x7f39dc1457f0 2024-03-29T11:22:03.780 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.793+0000 7f39d1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:03.780 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.793+0000 7f39d1ffb640 10 client.?.objecter ms_handle_connect 0x7f39dc13f650 2024-03-29T11:22:03.780 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.793+0000 7f39d1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:03.781 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.793+0000 7f39d1ffb640 10 client.?.objecter ms_handle_connect 0x7f39dc0a9d70 2024-03-29T11:22:03.781 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.793+0000 7f39d1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:03.781 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.793+0000 7f39d1ffb640 1 -- 172.21.15.47:0/748854794 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2610421301 0 0) 0x7f39dc1cbff0 con 0x7f39dc1457f0 2024-03-29T11:22:03.781 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.793+0000 7f39d1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:03.781 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.793+0000 7f39d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:03.781 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.793+0000 7f39d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:03.781 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.793+0000 7f39d1ffb640 10 monclient(hunting): my global_id is 4522 2024-03-29T11:22:03.781 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.793+0000 7f39d1ffb640 1 -- 172.21.15.47:0/748854794 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39c0004eb0 con 0x7f39dc1457f0 2024-03-29T11:22:03.781 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.794+0000 7f39d1ffb640 1 -- 172.21.15.47:0/748854794 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2323756117 0 0) 0x7f39dc1404c0 con 0x7f39dc13f650 2024-03-29T11:22:03.781 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.794+0000 7f39d1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:03.781 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.794+0000 7f39d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:03.781 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.794+0000 7f39d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:03.781 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.794+0000 7f39d1ffb640 10 monclient(hunting): my global_id is 4601 2024-03-29T11:22:03.781 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.794+0000 7f39d1ffb640 1 -- 172.21.15.47:0/748854794 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39dc1cbff0 con 0x7f39dc13f650 2024-03-29T11:22:03.781 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.794+0000 7f39d1ffb640 1 -- 172.21.15.47:0/748854794 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1231180468 0 0) 0x7f39dc1cbd10 con 0x7f39dc0a9d70 2024-03-29T11:22:03.781 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.794+0000 7f39d1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:03.782 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.794+0000 7f39d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:03.782 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.794+0000 7f39d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:03.782 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.794+0000 7f39d1ffb640 10 monclient(hunting): my global_id is 4539 2024-03-29T11:22:03.782 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.794+0000 7f39d1ffb640 1 -- 172.21.15.47:0/748854794 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39dc1404c0 con 0x7f39dc0a9d70 2024-03-29T11:22:03.782 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.795+0000 7f39d1ffb640 1 -- 172.21.15.47:0/748854794 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4293378690 0 0) 0x7f39c0004eb0 con 0x7f39dc1457f0 2024-03-29T11:22:03.782 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.795+0000 7f39d1ffb640 1 -- 172.21.15.47:0/748854794 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f39dc1cbd10 con 0x7f39dc1457f0 2024-03-29T11:22:03.782 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.795+0000 7f39d1ffb640 1 -- 172.21.15.47:0/748854794 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f39d4001cd0 con 0x7f39dc1457f0 2024-03-29T11:22:03.782 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.795+0000 7f39d1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:03.782 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.795+0000 7f39d1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:03.782 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.795+0000 7f39d1ffb640 10 monclient(hunting): dump: 2024-03-29T11:22:03.782 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:22:03.782 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:22:03.782 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:03.782 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:03.783 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:03.783 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:22:03.783 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:22:03.783 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:22:03.783 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:22:03.783 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:22:03.783 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.795+0000 7f39d1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:03.783 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.795+0000 7f39d1ffb640 1 -- 172.21.15.47:0/748854794 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3540126423 0 0) 0x7f39dc1cbff0 con 0x7f39dc13f650 2024-03-29T11:22:03.783 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.795+0000 7f39d1ffb640 1 -- 172.21.15.47:0/748854794 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f39c0004eb0 con 0x7f39dc13f650 2024-03-29T11:22:03.783 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.795+0000 7f39d1ffb640 1 -- 172.21.15.47:0/748854794 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f39cc0018b0 con 0x7f39dc13f650 2024-03-29T11:22:03.783 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.795+0000 7f39d1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:03.783 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.795+0000 7f39d1ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:03.783 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.795+0000 7f39d1ffb640 10 monclient(hunting): dump: 2024-03-29T11:22:03.783 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:22:03.784 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:22:03.784 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:03.784 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:03.784 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:03.784 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:22:03.784 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:22:03.784 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:22:03.784 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:22:03.784 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:22:03.784 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.795+0000 7f39d1ffb640 1 -- 172.21.15.47:0/748854794 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2830028841 0 0) 0x7f39dc1cbd10 con 0x7f39dc1457f0 2024-03-29T11:22:03.784 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.795+0000 7f39d1ffb640 1 -- 172.21.15.47:0/748854794 >> v1:172.21.15.47:6790/0 conn(0x7f39dc13f650 legacy=0x7f39dc136d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:03.784 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.795+0000 7f39d1ffb640 1 -- 172.21.15.47:0/748854794 >> v1:172.21.15.47:6789/0 conn(0x7f39dc0a9d70 legacy=0x7f39dc141940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:03.784 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.795+0000 7f39d1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:22:03.784 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.795+0000 7f39d1ffb640 1 monclient: found mon.b 2024-03-29T11:22:03.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.795+0000 7f39d1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:03.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.795+0000 7f39d1ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:22:03.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.795+0000 7f39d1ffb640 1 -- 172.21.15.47:0/748854794 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39dc144860 con 0x7f39dc1457f0 2024-03-29T11:22:03.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.795+0000 7f39d1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:22:03.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.795+0000 7f39d1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:22:03.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.795+0000 7f39d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:03.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.795+0000 7f39e396b640 5 monclient: authenticate success, global_id 4522 2024-03-29T11:22:03.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.795+0000 7f39e396b640 10 monclient: _renew_subs 2024-03-29T11:22:03.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.795+0000 7f39e396b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:22:03.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.795+0000 7f39e396b640 1 -- 172.21.15.47:0/748854794 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39dc144b10 con 0x7f39dc1457f0 2024-03-29T11:22:03.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.796+0000 7f39e396b640 10 client.4522.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:03.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.796+0000 7f39e396b640 10 monclient: _renew_subs 2024-03-29T11:22:03.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.796+0000 7f39e396b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:22:03.785 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.796+0000 7f39e396b640 1 -- 172.21.15.47:0/748854794 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f39dc137620 con 0x7f39dc1457f0 2024-03-29T11:22:03.786 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.796+0000 7f39e396b640 1 librados: init done 2024-03-29T11:22:03.786 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.796+0000 7f39d1ffb640 1 -- 172.21.15.47:0/748854794 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f39d40020e0 con 0x7f39dc1457f0 2024-03-29T11:22:03.786 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.796+0000 7f39d1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:03.786 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.796+0000 7f39d1ffb640 1 -- 172.21.15.47:0/748854794 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f39d4004d40 con 0x7f39dc1457f0 2024-03-29T11:22:03.786 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.796+0000 7f39d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:03.786 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.796+0000 7f39d1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:03.786 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.796+0000 7f39d1ffb640 10 monclient: dump: 2024-03-29T11:22:03.786 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:22:03.786 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:22:03.786 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:03.786 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:03.786 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:03.786 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:22:03.786 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:22:03.787 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:22:03.787 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:22:03.787 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:22:03.787 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.796+0000 7f39d37fe640 4 set_mon_vals no callback set 2024-03-29T11:22:03.787 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.797+0000 7f39e396b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:03.787 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.797+0000 7f39e396b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:03.787 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.797+0000 7f39e396b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:22:03.787 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.797+0000 7f39e396b640 1 -- 172.21.15.47:0/748854794 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39bc005080 con 0x7f39dc1457f0 2024-03-29T11:22:03.787 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.798+0000 7f39d1ffb640 1 -- 172.21.15.47:0/748854794 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7f39d4004ea0 con 0x7f39dc1457f0 2024-03-29T11:22:03.787 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.799+0000 7f39d1ffb640 1 -- 172.21.15.47:0/748854794 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7f39d4084da0 con 0x7f39dc1457f0 2024-03-29T11:22:03.787 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.799+0000 7f39d1ffb640 10 client.4522.objecter ms_dispatch 0x7f39dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:03.787 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.799+0000 7f39d1ffb640 3 client.4522.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:03.787 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.799+0000 7f39d1ffb640 3 client.4522.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:03.788 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.799+0000 7f39d1ffb640 20 client.4522.objecter dump_active .. 0 homeless 2024-03-29T11:22:03.788 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.799+0000 7f39e1707640 10 monclient: get_auth_request con 0x7f39c006dfe0 auth_method 0 2024-03-29T11:22:03.788 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.799+0000 7f39d1ffb640 10 client.4522.objecter ms_handle_connect 0x7f39c006dfe0 2024-03-29T11:22:03.788 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.809+0000 7f39d1ffb640 1 -- 172.21.15.47:0/748854794 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f39d40504b0 con 0x7f39dc1457f0 2024-03-29T11:22:03.788 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.809+0000 7f39d1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:03.788 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:03.809+0000 7f39d1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:04.129 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:04.152+0000 7f39e396b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:04.129 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:04.152+0000 7f39e396b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:04.129 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:04.152+0000 7f39e396b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:22:04.129 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:04.152+0000 7f39e396b640 1 -- 172.21.15.47:0/748854794 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f39bc005330 con 0x7f39dc1457f0 2024-03-29T11:22:04.130 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:04.153+0000 7f39d1ffb640 1 -- 172.21.15.47:0/748854794 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7f39d40525b0 con 0x7f39dc1457f0 2024-03-29T11:22:04.130 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:04.153+0000 7f39d1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:04.130 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:04.153+0000 7f39d1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:04.130 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:22:04.131 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:04.134 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:04.156+0000 7f39e396b640 10 librados: watch_flush enter 2024-03-29T11:22:04.134 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:04.158+0000 7f39e396b640 10 librados: watch_flush exit 2024-03-29T11:22:04.134 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:04.158+0000 7f39e396b640 20 client.4522.objecter shutdown clearing up homeless session... 2024-03-29T11:22:04.134 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:04.158+0000 7f39e396b640 10 client.4522.objecter successfully canceled tick 2024-03-29T11:22:04.135 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:04.159+0000 7f39e396b640 1 -- 172.21.15.47:0/748854794 >> v1:172.21.15.121:6816/1964117370 conn(0x7f39c006dfe0 legacy=0x7f39c00704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:04.135 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:04.159+0000 7f39e396b640 10 monclient: shutdown 2024-03-29T11:22:04.136 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:04.159+0000 7f39e396b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:04.136 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:04.159+0000 7f39e396b640 1 -- 172.21.15.47:0/748854794 >> v1:172.21.15.121:6789/0 conn(0x7f39dc1457f0 legacy=0x7f39dc1e2af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:04.136 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:04.160+0000 7f39e396b640 1 -- 172.21.15.47:0/748854794 shutdown_connections 2024-03-29T11:22:04.136 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:04.160+0000 7f39e396b640 1 -- 172.21.15.47:0/748854794 >> 172.21.15.47:0/748854794 conn(0x7f39dc066a00 msgr2=0x7f39dc13a920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:04.138 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:04.162+0000 7f39e396b640 1 -- 172.21.15.47:0/748854794 shutdown_connections 2024-03-29T11:22:04.138 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:04.162+0000 7f39e396b640 1 -- 172.21.15.47:0/748854794 wait complete. 2024-03-29T11:22:04.138 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:04.162+0000 7f39e396b640 1 librados: shutdown 2024-03-29T11:22:04.161 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:10.968 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:11.132 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.155+0000 7fbd566d3640 10 monclient: get_monmap_and_config 2024-03-29T11:22:11.132 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.155+0000 7fbd566d3640 10 monclient: build_initial_monmap 2024-03-29T11:22:11.132 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.155+0000 7fbd566d3640 10 monclient: monmap: 2024-03-29T11:22:11.132 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:22:11.133 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:22:11.133 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:22:11.157090+0000 2024-03-29T11:22:11.133 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:22:11.157090+0000 2024-03-29T11:22:11.133 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:11.133 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:22:11.133 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:22:11.133 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:22:11.133 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:22:11.133 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:22:11.133 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.156+0000 7fbd566d3640 1 Processor -- start 2024-03-29T11:22:11.134 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.156+0000 7fbd566d3640 1 -- start start 2024-03-29T11:22:11.134 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.156+0000 7fbd566d3640 10 monclient: init 2024-03-29T11:22:11.134 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.157+0000 7fbd566d3640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:11.134 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.157+0000 7fbd566d3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:22:11.135 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.157+0000 7fbd566d3640 10 monclient(hunting): picked mon.noname-a con 0x7fbd50070170 addr v1:172.21.15.47:6789/0 2024-03-29T11:22:11.135 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.157+0000 7fbd566d3640 10 monclient(hunting): picked mon.noname-b con 0x7fbd500713f0 addr v1:172.21.15.47:6790/0 2024-03-29T11:22:11.135 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.157+0000 7fbd566d3640 10 monclient(hunting): picked mon.noname-c con 0x7fbd50070550 addr v1:172.21.15.121:6789/0 2024-03-29T11:22:11.135 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.157+0000 7fbd566d3640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd5013dbd0 con 0x7fbd50070170 2024-03-29T11:22:11.135 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.157+0000 7fbd566d3640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd5013ff70 con 0x7fbd50070550 2024-03-29T11:22:11.135 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.157+0000 7fbd566d3640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd50140250 con 0x7fbd500713f0 2024-03-29T11:22:11.135 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.157+0000 7fbd566d3640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:11.135 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.157+0000 7fbd566d3640 10 monclient(hunting): authenticate will time out at 1374.346543s 2024-03-29T11:22:11.136 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.157+0000 7fbd4f7fe640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fbd500713f0 0x7fbd501406e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:38790/0 (socket says 172.21.15.47:38790) 2024-03-29T11:22:11.137 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.157+0000 7fbd54c70640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fbd50070550 0x7fbd5014b7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.47:52976/0 (socket says 172.21.15.47:52976) 2024-03-29T11:22:11.137 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.157+0000 7fbd4f7fe640 1 -- 172.21.15.47:0/550531633 learned_addr learned my addr 172.21.15.47:0/550531633 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:22:11.137 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.158+0000 7fbd4effd640 1 -- 172.21.15.47:0/550531633 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4099057660 0 0) 0x7fbd40001040 con 0x7fbd50070170 2024-03-29T11:22:11.137 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.158+0000 7fbd4effd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:11.137 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.158+0000 7fbd4effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:11.137 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.158+0000 7fbd4effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:11.137 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.158+0000 7fbd4effd640 10 monclient(hunting): my global_id is 4542 2024-03-29T11:22:11.137 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.158+0000 7fbd4effd640 1 -- 172.21.15.47:0/550531633 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd38003520 con 0x7fbd50070170 2024-03-29T11:22:11.137 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.158+0000 7fbd4effd640 1 -- 172.21.15.47:0/550531633 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1636569815 0 0) 0x7fbd38003520 con 0x7fbd50070170 2024-03-29T11:22:11.137 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.158+0000 7fbd4effd640 1 -- 172.21.15.47:0/550531633 >> v1:172.21.15.47:6790/0 conn(0x7fbd500713f0 legacy=0x7fbd501406e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:11.137 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.158+0000 7fbd4effd640 1 -- 172.21.15.47:0/550531633 >> v1:172.21.15.121:6789/0 conn(0x7fbd50070550 legacy=0x7fbd5014b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:11.137 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.158+0000 7fbd4effd640 10 monclient: _finish_hunting 0 2024-03-29T11:22:11.137 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.158+0000 7fbd4effd640 1 monclient: found mon.noname-a 2024-03-29T11:22:11.137 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.158+0000 7fbd4effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:11.138 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.158+0000 7fbd4effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-29T11:22:11.138 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.158+0000 7fbd4effd640 1 -- 172.21.15.47:0/550531633 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd50140df0 con 0x7fbd50070170 2024-03-29T11:22:11.138 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.158+0000 7fbd4effd640 10 monclient: _finish_auth 0 2024-03-29T11:22:11.138 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.158+0000 7fbd4effd640 10 monclient: _check_auth_tickets 2024-03-29T11:22:11.138 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.158+0000 7fbd4effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:11.138 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.158+0000 7fbd566d3640 5 monclient: authenticate success, global_id 4542 2024-03-29T11:22:11.138 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.158+0000 7fbd566d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:11.138 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.158+0000 7fbd4effd640 1 -- 172.21.15.47:0/550531633 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fbd400015b0 con 0x7fbd50070170 2024-03-29T11:22:11.138 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.158+0000 7fbd4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:11.138 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.158+0000 7fbd4effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:11.138 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.158+0000 7fbd4effd640 10 monclient: dump: 2024-03-29T11:22:11.138 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:22:11.139 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:22:11.139 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:11.139 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:11.139 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:11.139 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:22:11.139 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:22:11.139 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:22:11.139 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:22:11.139 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:22:11.139 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.158+0000 7fbd566d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:11.139 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.158+0000 7fbd4effd640 1 -- 172.21.15.47:0/550531633 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd40001840 con 0x7fbd50070170 2024-03-29T11:22:11.139 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.158+0000 7fbd4effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:11.139 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.158+0000 7fbd4effd640 1 -- 172.21.15.47:0/550531633 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fbd40001bd0 con 0x7fbd50070170 2024-03-29T11:22:11.140 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.158+0000 7fbd4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:11.140 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.158+0000 7fbd4effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:11.140 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.158+0000 7fbd4effd640 10 monclient: dump: 2024-03-29T11:22:11.140 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:22:11.140 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:22:11.140 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:11.140 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:11.140 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:11.140 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:22:11.140 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:22:11.140 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:22:11.140 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:22:11.140 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:22:11.140 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.158+0000 7fbd566d3640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:11.141 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.158+0000 7fbd566d3640 4 set_mon_vals no callback set 2024-03-29T11:22:11.141 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.159+0000 7fbd566d3640 10 monclient: shutdown 2024-03-29T11:22:11.141 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.159+0000 7fbd566d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:11.141 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.159+0000 7fbd566d3640 1 -- 172.21.15.47:0/550531633 >> v1:172.21.15.47:6789/0 conn(0x7fbd50070170 legacy=0x7fbd5013d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:11.141 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.159+0000 7fbd566d3640 1 -- 172.21.15.47:0/550531633 shutdown_connections 2024-03-29T11:22:11.141 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.159+0000 7fbd566d3640 1 -- 172.21.15.47:0/550531633 >> 172.21.15.47:0/550531633 conn(0x7fbd50067610 msgr2=0x7fbd500b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:11.144 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.159+0000 7fbd566d3640 1 -- 172.21.15.47:0/550531633 shutdown_connections 2024-03-29T11:22:11.144 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.159+0000 7fbd566d3640 1 -- 172.21.15.47:0/550531633 wait complete. 2024-03-29T11:22:11.144 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.160+0000 7fbd566d3640 10 monclient: build_initial_monmap 2024-03-29T11:22:11.144 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.160+0000 7fbd566d3640 10 monclient: monmap: 2024-03-29T11:22:11.145 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:22:11.145 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:11.145 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:22:11.145 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:22:11.145 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:11.145 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:22:11.145 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:22:11.145 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:22:11.145 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:22:11.145 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:22:11.145 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.161+0000 7fbd566d3640 1 librados: starting msgr at 2024-03-29T11:22:11.145 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.161+0000 7fbd566d3640 1 librados: starting objecter 2024-03-29T11:22:11.145 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.161+0000 7fbd566d3640 1 Processor -- start 2024-03-29T11:22:11.145 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.161+0000 7fbd566d3640 1 -- start start 2024-03-29T11:22:11.146 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.161+0000 7fbd566d3640 1 librados: setting wanted keys 2024-03-29T11:22:11.146 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.161+0000 7fbd566d3640 1 librados: calling monclient init 2024-03-29T11:22:11.146 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.161+0000 7fbd566d3640 10 monclient: init 2024-03-29T11:22:11.146 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.162+0000 7fbd566d3640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:11.146 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.162+0000 7fbd566d3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:22:11.146 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.162+0000 7fbd566d3640 10 monclient(hunting): picked mon.noname-b con 0x7fbd50070170 addr v1:172.21.15.121:6789/0 2024-03-29T11:22:11.146 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.162+0000 7fbd566d3640 10 monclient(hunting): picked mon.noname-c con 0x7fbd50070550 addr v1:172.21.15.47:6790/0 2024-03-29T11:22:11.146 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.162+0000 7fbd566d3640 10 monclient(hunting): picked mon.noname-a con 0x7fbd500713f0 addr v1:172.21.15.47:6789/0 2024-03-29T11:22:11.146 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.162+0000 7fbd54c70640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fbd500713f0 0x7fbd501df470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:43166/0 (socket says 172.21.15.47:43166) 2024-03-29T11:22:11.146 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.162+0000 7fbd54c70640 1 -- 172.21.15.47:0/758165402 learned_addr learned my addr 172.21.15.47:0/758165402 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:22:11.146 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.162+0000 7fbd566d3640 1 -- 172.21.15.47:0/758165402 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd50138b20 con 0x7fbd500713f0 2024-03-29T11:22:11.146 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.162+0000 7fbd4cff9640 10 client.?.objecter ms_handle_connect 0x7fbd500713f0 2024-03-29T11:22:11.146 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.162+0000 7fbd4cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:11.146 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.163+0000 7fbd566d3640 1 -- 172.21.15.47:0/758165402 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd501dfb80 con 0x7fbd50070170 2024-03-29T11:22:11.147 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.163+0000 7fbd566d3640 1 -- 172.21.15.47:0/758165402 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd5013a500 con 0x7fbd50070550 2024-03-29T11:22:11.147 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.163+0000 7fbd566d3640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:11.147 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.163+0000 7fbd566d3640 10 monclient(hunting): authenticate will time out at 1374.352024s 2024-03-29T11:22:11.147 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.163+0000 7fbd4cff9640 10 client.?.objecter ms_handle_connect 0x7fbd50070170 2024-03-29T11:22:11.147 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.163+0000 7fbd4cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:11.147 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.163+0000 7fbd4cff9640 1 -- 172.21.15.47:0/758165402 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3678690009 0 0) 0x7fbd50138b20 con 0x7fbd500713f0 2024-03-29T11:22:11.147 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.163+0000 7fbd4cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:22:11.147 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.163+0000 7fbd4cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:11.147 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.163+0000 7fbd4cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:11.147 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.163+0000 7fbd4cff9640 10 monclient(hunting): my global_id is 4545 2024-03-29T11:22:11.147 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.163+0000 7fbd4cff9640 1 -- 172.21.15.47:0/758165402 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd3c004eb0 con 0x7fbd500713f0 2024-03-29T11:22:11.148 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.164+0000 7fbd4cff9640 10 client.?.objecter ms_handle_connect 0x7fbd50070550 2024-03-29T11:22:11.148 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.164+0000 7fbd4cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:11.148 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.164+0000 7fbd4cff9640 1 -- 172.21.15.47:0/758165402 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3367107377 0 0) 0x7fbd3c004eb0 con 0x7fbd500713f0 2024-03-29T11:22:11.148 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.164+0000 7fbd4cff9640 1 -- 172.21.15.47:0/758165402 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd50138b20 con 0x7fbd500713f0 2024-03-29T11:22:11.148 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.164+0000 7fbd4cff9640 1 -- 172.21.15.47:0/758165402 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fbd48001c30 con 0x7fbd500713f0 2024-03-29T11:22:11.148 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.164+0000 7fbd4cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:11.148 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.164+0000 7fbd4cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:11.148 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.164+0000 7fbd4cff9640 10 monclient(hunting): dump: 2024-03-29T11:22:11.148 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:22:11.148 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:22:11.148 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:11.148 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:11.148 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:11.148 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:22:11.148 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:22:11.149 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:22:11.149 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:22:11.149 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:22:11.149 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.164+0000 7fbd4cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:11.149 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.164+0000 7fbd4cff9640 1 -- 172.21.15.47:0/758165402 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 987300304 0 0) 0x7fbd5013a500 con 0x7fbd50070550 2024-03-29T11:22:11.149 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.164+0000 7fbd4cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:22:11.149 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.164+0000 7fbd4cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:11.149 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.164+0000 7fbd4cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:11.149 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.164+0000 7fbd4cff9640 10 monclient(hunting): my global_id is 4607 2024-03-29T11:22:11.149 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.164+0000 7fbd4cff9640 1 -- 172.21.15.47:0/758165402 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd3c004eb0 con 0x7fbd50070550 2024-03-29T11:22:11.149 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.164+0000 7fbd4cff9640 1 -- 172.21.15.47:0/758165402 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3914939498 0 0) 0x7fbd50138b20 con 0x7fbd500713f0 2024-03-29T11:22:11.149 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.164+0000 7fbd4cff9640 1 -- 172.21.15.47:0/758165402 >> v1:172.21.15.47:6790/0 conn(0x7fbd50070550 legacy=0x7fbd501ded60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:11.149 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.164+0000 7fbd4cff9640 1 -- 172.21.15.47:0/758165402 >> v1:172.21.15.121:6789/0 conn(0x7fbd50070170 legacy=0x7fbd5012ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:11.149 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.164+0000 7fbd4cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:22:11.149 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.164+0000 7fbd4cff9640 1 monclient: found mon.a 2024-03-29T11:22:11.150 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.164+0000 7fbd4cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:11.150 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.164+0000 7fbd4cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:22:11.150 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.164+0000 7fbd4cff9640 1 -- 172.21.15.47:0/758165402 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd5013a7e0 con 0x7fbd500713f0 2024-03-29T11:22:11.150 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.164+0000 7fbd4cff9640 10 monclient: _finish_auth 0 2024-03-29T11:22:11.150 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.164+0000 7fbd4cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:22:11.150 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.164+0000 7fbd4cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:11.150 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.164+0000 7fbd566d3640 5 monclient: authenticate success, global_id 4545 2024-03-29T11:22:11.150 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.165+0000 7fbd566d3640 10 monclient: _renew_subs 2024-03-29T11:22:11.150 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.165+0000 7fbd566d3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:22:11.150 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.165+0000 7fbd566d3640 1 -- 172.21.15.47:0/758165402 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd5013aac0 con 0x7fbd500713f0 2024-03-29T11:22:11.150 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.165+0000 7fbd566d3640 10 client.4545.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:11.150 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.165+0000 7fbd566d3640 10 monclient: _renew_subs 2024-03-29T11:22:11.150 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.165+0000 7fbd566d3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:22:11.150 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.165+0000 7fbd566d3640 1 -- 172.21.15.47:0/758165402 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd50131430 con 0x7fbd500713f0 2024-03-29T11:22:11.151 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.165+0000 7fbd566d3640 1 librados: init done 2024-03-29T11:22:11.151 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.165+0000 7fbd4cff9640 1 -- 172.21.15.47:0/758165402 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd48001d90 con 0x7fbd500713f0 2024-03-29T11:22:11.151 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.165+0000 7fbd4cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:11.151 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.165+0000 7fbd4cff9640 1 -- 172.21.15.47:0/758165402 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fbd48004dc0 con 0x7fbd500713f0 2024-03-29T11:22:11.151 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.165+0000 7fbd4cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:11.151 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.165+0000 7fbd4cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:11.151 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.165+0000 7fbd4cff9640 10 monclient: dump: 2024-03-29T11:22:11.151 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:22:11.151 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:22:11.151 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:11.151 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:11.152 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:11.152 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:22:11.152 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:22:11.152 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:22:11.152 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:22:11.152 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:22:11.152 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.165+0000 7fbd4e7fc640 4 set_mon_vals no callback set 2024-03-29T11:22:11.154 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.168+0000 7fbd566d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:11.155 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.168+0000 7fbd4cff9640 1 -- 172.21.15.47:0/758165402 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7fbd48004f20 con 0x7fbd500713f0 2024-03-29T11:22:11.155 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.168+0000 7fbd566d3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:11.155 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.168+0000 7fbd566d3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:22:11.155 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.168+0000 7fbd566d3640 1 -- 172.21.15.47:0/758165402 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd30005080 con 0x7fbd500713f0 2024-03-29T11:22:11.155 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.169+0000 7fbd4ffff640 10 monclient: get_auth_request con 0x7fbd3c06dd00 auth_method 0 2024-03-29T11:22:11.155 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.169+0000 7fbd4cff9640 1 -- 172.21.15.47:0/758165402 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7fbd48084db0 con 0x7fbd500713f0 2024-03-29T11:22:11.155 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.169+0000 7fbd4cff9640 10 client.4545.objecter ms_dispatch 0x7fbd50000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:11.155 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.169+0000 7fbd4cff9640 3 client.4545.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:11.156 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.169+0000 7fbd4cff9640 3 client.4545.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:11.156 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.169+0000 7fbd4cff9640 20 client.4545.objecter dump_active .. 0 homeless 2024-03-29T11:22:11.156 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.169+0000 7fbd4cff9640 10 client.4545.objecter ms_handle_connect 0x7fbd3c06dd00 2024-03-29T11:22:11.156 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.179+0000 7fbd4cff9640 1 -- 172.21.15.47:0/758165402 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbd480506f0 con 0x7fbd500713f0 2024-03-29T11:22:11.156 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.179+0000 7fbd4cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:11.157 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.179+0000 7fbd4cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:11.546 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.569+0000 7fbd566d3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:11.546 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.569+0000 7fbd566d3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:11.547 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.569+0000 7fbd566d3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:22:11.547 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.569+0000 7fbd566d3640 1 -- 172.21.15.47:0/758165402 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbd30005330 con 0x7fbd500713f0 2024-03-29T11:22:11.547 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.570+0000 7fbd4cff9640 1 -- 172.21.15.47:0/758165402 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7fbd48084460 con 0x7fbd500713f0 2024-03-29T11:22:11.547 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.570+0000 7fbd4cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:11.548 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.570+0000 7fbd4cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:11.548 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:22:11.548 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:11.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.573+0000 7fbd566d3640 10 librados: watch_flush enter 2024-03-29T11:22:11.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.574+0000 7fbd566d3640 10 librados: watch_flush exit 2024-03-29T11:22:11.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.574+0000 7fbd566d3640 20 client.4545.objecter shutdown clearing up homeless session... 2024-03-29T11:22:11.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.574+0000 7fbd566d3640 10 client.4545.objecter successfully canceled tick 2024-03-29T11:22:11.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.574+0000 7fbd566d3640 1 -- 172.21.15.47:0/758165402 >> v1:172.21.15.121:6816/1964117370 conn(0x7fbd3c06dd00 legacy=0x7fbd3c0701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:11.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.574+0000 7fbd566d3640 10 monclient: shutdown 2024-03-29T11:22:11.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.574+0000 7fbd566d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:11.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.574+0000 7fbd566d3640 1 -- 172.21.15.47:0/758165402 >> v1:172.21.15.47:6789/0 conn(0x7fbd500713f0 legacy=0x7fbd501df470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:11.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.574+0000 7fbd566d3640 1 -- 172.21.15.47:0/758165402 shutdown_connections 2024-03-29T11:22:11.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.574+0000 7fbd566d3640 1 -- 172.21.15.47:0/758165402 >> 172.21.15.47:0/758165402 conn(0x7fbd50067610 msgr2=0x7fbd50069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:11.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.574+0000 7fbd566d3640 1 -- 172.21.15.47:0/758165402 shutdown_connections 2024-03-29T11:22:11.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.574+0000 7fbd566d3640 1 -- 172.21.15.47:0/758165402 wait complete. 2024-03-29T11:22:11.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:11.574+0000 7fbd566d3640 1 librados: shutdown 2024-03-29T11:22:11.567 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:18.373 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:18.525 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.548+0000 7f70e21b6640 10 monclient: get_monmap_and_config 2024-03-29T11:22:18.525 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.548+0000 7f70e21b6640 10 monclient: build_initial_monmap 2024-03-29T11:22:18.526 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.548+0000 7f70e21b6640 10 monclient: monmap: 2024-03-29T11:22:18.526 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:22:18.526 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:22:18.526 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:22:18.549923+0000 2024-03-29T11:22:18.526 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:22:18.549923+0000 2024-03-29T11:22:18.526 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:18.526 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:22:18.526 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:22:18.526 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:22:18.527 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:22:18.527 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:22:18.527 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.550+0000 7f70e21b6640 1 Processor -- start 2024-03-29T11:22:18.529 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.550+0000 7f70e21b6640 1 -- start start 2024-03-29T11:22:18.529 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.550+0000 7f70e21b6640 10 monclient: init 2024-03-29T11:22:18.529 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.551+0000 7f70e21b6640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:18.529 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.551+0000 7f70e21b6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:22:18.529 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.551+0000 7f70e21b6640 10 monclient(hunting): picked mon.noname-b con 0x7f70dc0b0770 addr v1:172.21.15.47:6790/0 2024-03-29T11:22:18.529 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.551+0000 7f70e21b6640 10 monclient(hunting): picked mon.noname-a con 0x7f70dc14a700 addr v1:172.21.15.47:6789/0 2024-03-29T11:22:18.529 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.551+0000 7f70e21b6640 10 monclient(hunting): picked mon.noname-c con 0x7f70dc0ac5a0 addr v1:172.21.15.121:6789/0 2024-03-29T11:22:18.529 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.551+0000 7f70e21b6640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70dc0aba70 con 0x7f70dc14a700 2024-03-29T11:22:18.529 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.551+0000 7f70e21b6640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70dc0aecb0 con 0x7f70dc0ac5a0 2024-03-29T11:22:18.530 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.551+0000 7f70e21b6640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70dc0af150 con 0x7f70dc0b0770 2024-03-29T11:22:18.530 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.551+0000 7f70e21b6640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:18.530 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.551+0000 7f70e21b6640 10 monclient(hunting): authenticate will time out at 1381.739120s 2024-03-29T11:22:18.533 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.557+0000 7f70db7fe640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f70dc0b0770 0x7f70dc149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:57121/0 (socket says 172.21.15.47:57121) 2024-03-29T11:22:18.533 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.557+0000 7f70db7fe640 1 -- 172.21.15.47:0/2189018868 learned_addr learned my addr 172.21.15.47:0/2189018868 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:22:18.537 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.560+0000 7f70da7fc640 1 -- 172.21.15.47:0/2189018868 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 301907317 0 0) 0x7f70dc0aecb0 con 0x7f70dc0ac5a0 2024-03-29T11:22:18.537 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.560+0000 7f70da7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:18.537 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.560+0000 7f70da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:18.537 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.560+0000 7f70da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:18.537 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.560+0000 7f70da7fc640 10 monclient(hunting): my global_id is 4528 2024-03-29T11:22:18.537 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.560+0000 7f70da7fc640 1 -- 172.21.15.47:0/2189018868 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70c4003520 con 0x7f70dc0ac5a0 2024-03-29T11:22:18.537 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.560+0000 7f70da7fc640 1 -- 172.21.15.47:0/2189018868 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1031806474 0 0) 0x7f70dc0aba70 con 0x7f70dc14a700 2024-03-29T11:22:18.537 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.560+0000 7f70da7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:18.537 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.560+0000 7f70da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:18.538 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.560+0000 7f70da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:18.538 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.560+0000 7f70da7fc640 10 monclient(hunting): my global_id is 4548 2024-03-29T11:22:18.538 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.560+0000 7f70da7fc640 1 -- 172.21.15.47:0/2189018868 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70dc0aecb0 con 0x7f70dc14a700 2024-03-29T11:22:18.538 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.561+0000 7f70da7fc640 1 -- 172.21.15.47:0/2189018868 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2701177805 0 0) 0x7f70dc0aecb0 con 0x7f70dc14a700 2024-03-29T11:22:18.539 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.561+0000 7f70da7fc640 1 -- 172.21.15.47:0/2189018868 >> v1:172.21.15.47:6790/0 conn(0x7f70dc0b0770 legacy=0x7f70dc149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:18.539 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.561+0000 7f70da7fc640 1 -- 172.21.15.47:0/2189018868 >> v1:172.21.15.121:6789/0 conn(0x7f70dc0ac5a0 legacy=0x7f70dc151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:18.539 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.561+0000 7f70da7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:22:18.539 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.561+0000 7f70da7fc640 1 monclient: found mon.noname-a 2024-03-29T11:22:18.539 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.561+0000 7f70da7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:18.539 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.561+0000 7f70da7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-29T11:22:18.539 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.561+0000 7f70da7fc640 1 -- 172.21.15.47:0/2189018868 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70dc0ac9d0 con 0x7f70dc14a700 2024-03-29T11:22:18.539 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.561+0000 7f70da7fc640 10 monclient: _finish_auth 0 2024-03-29T11:22:18.539 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.561+0000 7f70da7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:22:18.539 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.561+0000 7f70da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:18.539 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.561+0000 7f70da7fc640 1 -- 172.21.15.47:0/2189018868 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f70d0001520 con 0x7f70dc14a700 2024-03-29T11:22:18.539 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.561+0000 7f70da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:18.539 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.561+0000 7f70da7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:18.540 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.561+0000 7f70da7fc640 10 monclient: dump: 2024-03-29T11:22:18.540 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:22:18.540 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:22:18.540 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:18.540 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:18.540 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:18.540 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:22:18.540 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:22:18.540 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:22:18.540 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:22:18.540 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:22:18.540 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.561+0000 7f70e21b6640 5 monclient: authenticate success, global_id 4548 2024-03-29T11:22:18.540 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.561+0000 7f70e21b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:18.540 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.562+0000 7f70da7fc640 1 -- 172.21.15.47:0/2189018868 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f70d0001680 con 0x7f70dc14a700 2024-03-29T11:22:18.540 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.562+0000 7f70da7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:18.540 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.562+0000 7f70e21b6640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:18.541 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.562+0000 7f70e21b6640 4 set_mon_vals no callback set 2024-03-29T11:22:18.541 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.562+0000 7f70da7fc640 1 -- 172.21.15.47:0/2189018868 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f70d0001ad0 con 0x7f70dc14a700 2024-03-29T11:22:18.541 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.562+0000 7f70da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:18.541 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.562+0000 7f70da7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:18.541 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.562+0000 7f70da7fc640 10 monclient: dump: 2024-03-29T11:22:18.541 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:22:18.541 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:22:18.541 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:18.541 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:18.541 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:18.541 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:22:18.541 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:22:18.541 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:22:18.541 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:22:18.541 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:22:18.542 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.563+0000 7f70e21b6640 10 monclient: shutdown 2024-03-29T11:22:18.542 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.563+0000 7f70e21b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:18.542 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.563+0000 7f70e21b6640 1 -- 172.21.15.47:0/2189018868 >> v1:172.21.15.47:6789/0 conn(0x7f70dc14a700 legacy=0x7f70dc0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:18.545 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.568+0000 7f70e21b6640 1 -- 172.21.15.47:0/2189018868 shutdown_connections 2024-03-29T11:22:18.545 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.568+0000 7f70e21b6640 1 -- 172.21.15.47:0/2189018868 >> 172.21.15.47:0/2189018868 conn(0x7f70dc06b660 msgr2=0x7f70dc06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:18.545 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.569+0000 7f70e21b6640 1 -- 172.21.15.47:0/2189018868 shutdown_connections 2024-03-29T11:22:18.547 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.569+0000 7f70e21b6640 1 -- 172.21.15.47:0/2189018868 wait complete. 2024-03-29T11:22:18.547 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.570+0000 7f70e21b6640 10 monclient: build_initial_monmap 2024-03-29T11:22:18.547 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.570+0000 7f70e21b6640 10 monclient: monmap: 2024-03-29T11:22:18.547 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:22:18.547 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:18.547 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:22:18.548 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:22:18.548 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:18.548 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:22:18.548 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:22:18.548 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:22:18.548 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:22:18.548 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:22:18.548 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.570+0000 7f70e21b6640 1 librados: starting msgr at 2024-03-29T11:22:18.548 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.570+0000 7f70e21b6640 1 librados: starting objecter 2024-03-29T11:22:18.548 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.570+0000 7f70e21b6640 1 Processor -- start 2024-03-29T11:22:18.548 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.571+0000 7f70e21b6640 1 -- start start 2024-03-29T11:22:18.548 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.571+0000 7f70e21b6640 1 librados: setting wanted keys 2024-03-29T11:22:18.548 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.571+0000 7f70e21b6640 1 librados: calling monclient init 2024-03-29T11:22:18.548 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.571+0000 7f70e21b6640 10 monclient: init 2024-03-29T11:22:18.549 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.571+0000 7f70e21b6640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:18.550 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.571+0000 7f70e21b6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:22:18.550 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.571+0000 7f70e21b6640 10 monclient(hunting): picked mon.noname-b con 0x7f70dc0ac5a0 addr v1:172.21.15.121:6789/0 2024-03-29T11:22:18.550 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.571+0000 7f70e21b6640 10 monclient(hunting): picked mon.noname-c con 0x7f70dc0b0770 addr v1:172.21.15.47:6790/0 2024-03-29T11:22:18.550 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.571+0000 7f70e21b6640 10 monclient(hunting): picked mon.noname-a con 0x7f70dc14a700 addr v1:172.21.15.47:6789/0 2024-03-29T11:22:18.550 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.571+0000 7f70e21b6640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70dc1400f0 con 0x7f70dc14a700 2024-03-29T11:22:18.550 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.571+0000 7f70e21b6640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70dc1222a0 con 0x7f70dc0ac5a0 2024-03-29T11:22:18.550 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.571+0000 7f70e21b6640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70dc123140 con 0x7f70dc0b0770 2024-03-29T11:22:18.550 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.571+0000 7f70e21b6640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:18.550 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.571+0000 7f70e21b6640 10 monclient(hunting): authenticate will time out at 1381.759756s 2024-03-29T11:22:18.557 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.578+0000 7f70dbfff640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f70dc14a700 0x7f70dc1ffb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:47041/0 (socket says 172.21.15.47:47041) 2024-03-29T11:22:18.557 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.578+0000 7f70dbfff640 1 -- 172.21.15.47:0/3199951835 learned_addr learned my addr 172.21.15.47:0/3199951835 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:22:18.557 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.578+0000 7f70c3fff640 10 client.?.objecter ms_handle_connect 0x7f70dc14a700 2024-03-29T11:22:18.557 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.578+0000 7f70c3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:18.557 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.578+0000 7f70c3fff640 10 client.?.objecter ms_handle_connect 0x7f70dc0b0770 2024-03-29T11:22:18.557 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.578+0000 7f70c3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:18.557 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.578+0000 7f70c3fff640 1 -- 172.21.15.47:0/3199951835 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3650676260 0 0) 0x7f70dc1400f0 con 0x7f70dc14a700 2024-03-29T11:22:18.557 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.578+0000 7f70c3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:18.558 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.578+0000 7f70c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:18.558 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.578+0000 7f70c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:18.558 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.578+0000 7f70c3fff640 10 monclient(hunting): my global_id is 4551 2024-03-29T11:22:18.558 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.578+0000 7f70c3fff640 1 -- 172.21.15.47:0/3199951835 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70c8004eb0 con 0x7f70dc14a700 2024-03-29T11:22:18.558 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.578+0000 7f70c3fff640 1 -- 172.21.15.47:0/3199951835 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2769121383 0 0) 0x7f70dc123140 con 0x7f70dc0b0770 2024-03-29T11:22:18.558 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.578+0000 7f70c3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:18.558 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.578+0000 7f70c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:18.558 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.578+0000 7f70c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:18.558 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.578+0000 7f70c3fff640 10 monclient(hunting): my global_id is 4613 2024-03-29T11:22:18.558 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.578+0000 7f70c3fff640 1 -- 172.21.15.47:0/3199951835 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70dc1400f0 con 0x7f70dc0b0770 2024-03-29T11:22:18.558 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.579+0000 7f70c3fff640 1 -- 172.21.15.47:0/3199951835 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4013123981 0 0) 0x7f70c8004eb0 con 0x7f70dc14a700 2024-03-29T11:22:18.558 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.579+0000 7f70c3fff640 1 -- 172.21.15.47:0/3199951835 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f70dc123140 con 0x7f70dc14a700 2024-03-29T11:22:18.559 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.579+0000 7f70c3fff640 1 -- 172.21.15.47:0/3199951835 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f70d4001cf0 con 0x7f70dc14a700 2024-03-29T11:22:18.559 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.579+0000 7f70c3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:18.559 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.579+0000 7f70c3fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:18.559 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.579+0000 7f70c3fff640 10 monclient(hunting): dump: 2024-03-29T11:22:18.559 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:22:18.559 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:22:18.559 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:18.559 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:18.559 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:18.559 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:22:18.559 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:22:18.559 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:22:18.559 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:22:18.559 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:22:18.560 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.579+0000 7f70c3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:18.560 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.579+0000 7f70c3fff640 1 -- 172.21.15.47:0/3199951835 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4017226966 0 0) 0x7f70dc1400f0 con 0x7f70dc0b0770 2024-03-29T11:22:18.560 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.579+0000 7f70c3fff640 1 -- 172.21.15.47:0/3199951835 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f70c8004eb0 con 0x7f70dc0b0770 2024-03-29T11:22:18.560 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.579+0000 7f70c3fff640 1 -- 172.21.15.47:0/3199951835 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f70d0001a90 con 0x7f70dc0b0770 2024-03-29T11:22:18.560 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.579+0000 7f70c3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:18.560 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.579+0000 7f70c3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:18.560 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.579+0000 7f70c3fff640 10 monclient(hunting): dump: 2024-03-29T11:22:18.560 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:22:18.560 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:22:18.560 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:18.560 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:18.560 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:18.560 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:22:18.561 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:22:18.561 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:22:18.561 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:22:18.561 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:22:18.561 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.579+0000 7f70c3fff640 1 -- 172.21.15.47:0/3199951835 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3843431274 0 0) 0x7f70dc123140 con 0x7f70dc14a700 2024-03-29T11:22:18.561 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.579+0000 7f70c3fff640 1 -- 172.21.15.47:0/3199951835 >> v1:172.21.15.47:6790/0 conn(0x7f70dc0b0770 legacy=0x7f70dc1405d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:18.561 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.579+0000 7f70c3fff640 1 -- 172.21.15.47:0/3199951835 >> v1:172.21.15.121:6789/0 conn(0x7f70dc0ac5a0 legacy=0x7f70dc122a30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:22:18.563 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.583+0000 7f70c3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:22:18.563 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.583+0000 7f70c3fff640 1 monclient: found mon.a 2024-03-29T11:22:18.563 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.583+0000 7f70c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:18.563 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.583+0000 7f70c3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:22:18.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.583+0000 7f70c3fff640 1 -- 172.21.15.47:0/3199951835 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70dc140ce0 con 0x7f70dc14a700 2024-03-29T11:22:18.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.583+0000 7f70c3fff640 10 monclient: _finish_auth 0 2024-03-29T11:22:18.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.583+0000 7f70c3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:22:18.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.583+0000 7f70c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:18.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.583+0000 7f70e21b6640 5 monclient: authenticate success, global_id 4551 2024-03-29T11:22:18.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.583+0000 7f70e21b6640 10 monclient: _renew_subs 2024-03-29T11:22:18.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.583+0000 7f70e21b6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:22:18.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.583+0000 7f70e21b6640 1 -- 172.21.15.47:0/3199951835 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70dc11fc40 con 0x7f70dc14a700 2024-03-29T11:22:18.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.583+0000 7f70e21b6640 10 client.4551.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:18.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.583+0000 7f70e21b6640 10 monclient: _renew_subs 2024-03-29T11:22:18.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.583+0000 7f70e21b6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:22:18.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.583+0000 7f70e21b6640 1 -- 172.21.15.47:0/3199951835 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f70dc1db020 con 0x7f70dc14a700 2024-03-29T11:22:18.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.583+0000 7f70e21b6640 1 librados: init done 2024-03-29T11:22:18.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.585+0000 7f70e21b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:18.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.585+0000 7f70e21b6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:18.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.585+0000 7f70e21b6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:22:18.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.585+0000 7f70e21b6640 1 -- 172.21.15.47:0/3199951835 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70dc0ab560 con 0x7f70dc14a700 2024-03-29T11:22:18.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.588+0000 7f70c3fff640 1 -- 172.21.15.47:0/3199951835 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f70d4001e50 con 0x7f70dc14a700 2024-03-29T11:22:18.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.588+0000 7f70c3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:18.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.588+0000 7f70c3fff640 1 -- 172.21.15.47:0/3199951835 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f70d4003c40 con 0x7f70dc14a700 2024-03-29T11:22:18.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.588+0000 7f70c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:18.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.588+0000 7f70c3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:18.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.588+0000 7f70c3fff640 10 monclient: dump: 2024-03-29T11:22:18.566 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:22:18.566 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:22:18.566 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:18.566 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:18.566 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:18.566 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:22:18.566 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:22:18.566 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:22:18.566 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:22:18.566 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:22:18.566 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.588+0000 7f70d9ffb640 4 set_mon_vals no callback set 2024-03-29T11:22:18.568 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.590+0000 7f70c3fff640 1 -- 172.21.15.47:0/3199951835 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7f70d4003e60 con 0x7f70dc14a700 2024-03-29T11:22:18.568 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.591+0000 7f70c3fff640 1 -- 172.21.15.47:0/3199951835 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7f70d4084da0 con 0x7f70dc14a700 2024-03-29T11:22:18.568 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.591+0000 7f70c3fff640 10 client.4551.objecter ms_dispatch 0x7f70dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:18.568 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.591+0000 7f70c3fff640 3 client.4551.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:18.568 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.591+0000 7f70c3fff640 3 client.4551.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:18.568 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.591+0000 7f70c3fff640 20 client.4551.objecter dump_active .. 0 homeless 2024-03-29T11:22:18.568 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.591+0000 7f70db7fe640 10 monclient: get_auth_request con 0x7f70c806e020 auth_method 0 2024-03-29T11:22:18.568 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.592+0000 7f70c3fff640 10 client.4551.objecter ms_handle_connect 0x7f70c806e020 2024-03-29T11:22:18.574 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.597+0000 7f70c3fff640 1 -- 172.21.15.47:0/3199951835 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f70d4050680 con 0x7f70dc14a700 2024-03-29T11:22:18.574 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.597+0000 7f70c3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:18.574 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.597+0000 7f70c3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:18.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.911+0000 7f70e21b6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:18.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.912+0000 7f70e21b6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:18.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.912+0000 7f70e21b6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:22:18.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.912+0000 7f70e21b6640 1 -- 172.21.15.47:0/3199951835 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f70dc075f10 con 0x7f70dc14a700 2024-03-29T11:22:18.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.916+0000 7f70c3fff640 1 -- 172.21.15.47:0/3199951835 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7f70d40525b0 con 0x7f70dc14a700 2024-03-29T11:22:18.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.916+0000 7f70c3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:18.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.916+0000 7f70c3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:18.895 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:22:18.896 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:18.897 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.919+0000 7f70e21b6640 10 librados: watch_flush enter 2024-03-29T11:22:18.897 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.919+0000 7f70e21b6640 10 librados: watch_flush exit 2024-03-29T11:22:18.897 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.919+0000 7f70e21b6640 20 client.4551.objecter shutdown clearing up homeless session... 2024-03-29T11:22:18.897 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.919+0000 7f70e21b6640 10 client.4551.objecter successfully canceled tick 2024-03-29T11:22:18.897 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.920+0000 7f70e21b6640 1 -- 172.21.15.47:0/3199951835 >> v1:172.21.15.121:6816/1964117370 conn(0x7f70c806e020 legacy=0x7f70c80704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:18.897 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.920+0000 7f70e21b6640 10 monclient: shutdown 2024-03-29T11:22:18.897 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.920+0000 7f70e21b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:18.897 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.920+0000 7f70e21b6640 1 -- 172.21.15.47:0/3199951835 >> v1:172.21.15.47:6789/0 conn(0x7f70dc14a700 legacy=0x7f70dc1ffb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:18.897 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.920+0000 7f70e21b6640 1 -- 172.21.15.47:0/3199951835 shutdown_connections 2024-03-29T11:22:18.898 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.920+0000 7f70e21b6640 1 -- 172.21.15.47:0/3199951835 >> 172.21.15.47:0/3199951835 conn(0x7f70dc06b660 msgr2=0x7f70dc0b2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:18.898 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.920+0000 7f70e21b6640 1 -- 172.21.15.47:0/3199951835 shutdown_connections 2024-03-29T11:22:18.898 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.920+0000 7f70e21b6640 1 -- 172.21.15.47:0/3199951835 wait complete. 2024-03-29T11:22:18.898 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:18.920+0000 7f70e21b6640 1 librados: shutdown 2024-03-29T11:22:18.913 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:25.719 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:25.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.902+0000 7fd6a3fff640 10 monclient: get_monmap_and_config 2024-03-29T11:22:25.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.902+0000 7fd6a3fff640 10 monclient: build_initial_monmap 2024-03-29T11:22:25.879 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.902+0000 7fd6a3fff640 10 monclient: monmap: 2024-03-29T11:22:25.879 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:22:25.880 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:22:25.880 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:22:25.903848+0000 2024-03-29T11:22:25.880 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:22:25.903848+0000 2024-03-29T11:22:25.880 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:25.880 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:22:25.880 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:22:25.880 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:22:25.880 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:22:25.880 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:22:25.882 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.906+0000 7fd6a3fff640 1 Processor -- start 2024-03-29T11:22:25.882 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.906+0000 7fd6a3fff640 1 -- start start 2024-03-29T11:22:25.882 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.906+0000 7fd6a3fff640 10 monclient: init 2024-03-29T11:22:25.886 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.907+0000 7fd6a3fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:25.886 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.907+0000 7fd6a3fff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:22:25.886 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.907+0000 7fd6a3fff640 10 monclient(hunting): picked mon.noname-b con 0x7fd69c02b4c0 addr v1:172.21.15.47:6790/0 2024-03-29T11:22:25.886 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.907+0000 7fd6a3fff640 10 monclient(hunting): picked mon.noname-c con 0x7fd69c00b8a0 addr v1:172.21.15.121:6789/0 2024-03-29T11:22:25.886 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.907+0000 7fd6a3fff640 10 monclient(hunting): picked mon.noname-a con 0x7fd69c02a5a0 addr v1:172.21.15.47:6789/0 2024-03-29T11:22:25.886 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.907+0000 7fd6a3fff640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd69c02c5c0 con 0x7fd69c02a5a0 2024-03-29T11:22:25.886 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.907+0000 7fd6a3fff640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd69c0c11a0 con 0x7fd69c00b8a0 2024-03-29T11:22:25.886 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.907+0000 7fd6a3fff640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd69c02afa0 con 0x7fd69c02b4c0 2024-03-29T11:22:25.886 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.907+0000 7fd6a3fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:25.886 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.907+0000 7fd6a3fff640 10 monclient(hunting): authenticate will time out at 1389.095138s 2024-03-29T11:22:25.887 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.911+0000 7fd6a37fe640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fd69c02a5a0 0x7fd69c0c6d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:33055/0 (socket says 172.21.15.47:33055) 2024-03-29T11:22:25.887 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.911+0000 7fd6a37fe640 1 -- 172.21.15.47:0/1561224548 learned_addr learned my addr 172.21.15.47:0/1561224548 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:22:25.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.912+0000 7fd6a1ffb640 1 -- 172.21.15.47:0/1561224548 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4007466056 0 0) 0x7fd69c0c11a0 con 0x7fd69c00b8a0 2024-03-29T11:22:25.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.912+0000 7fd6a1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:25.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.912+0000 7fd6a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:25.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.912+0000 7fd6a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:25.889 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.912+0000 7fd6a1ffb640 10 monclient(hunting): my global_id is 4531 2024-03-29T11:22:25.889 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.912+0000 7fd6a1ffb640 1 -- 172.21.15.47:0/1561224548 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd68c003520 con 0x7fd69c00b8a0 2024-03-29T11:22:25.889 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.912+0000 7fd6a1ffb640 1 -- 172.21.15.47:0/1561224548 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3356177557 0 0) 0x7fd69c02c5c0 con 0x7fd69c02a5a0 2024-03-29T11:22:25.889 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.912+0000 7fd6a1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:25.889 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.912+0000 7fd6a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:25.889 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.912+0000 7fd6a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:25.889 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.912+0000 7fd6a1ffb640 10 monclient(hunting): my global_id is 4554 2024-03-29T11:22:25.889 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.912+0000 7fd6a1ffb640 1 -- 172.21.15.47:0/1561224548 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd69c0c11a0 con 0x7fd69c02a5a0 2024-03-29T11:22:25.889 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.913+0000 7fd6a1ffb640 1 -- 172.21.15.47:0/1561224548 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 149280902 0 0) 0x7fd69c0c11a0 con 0x7fd69c02a5a0 2024-03-29T11:22:25.889 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.913+0000 7fd6a1ffb640 1 -- 172.21.15.47:0/1561224548 >> v1:172.21.15.47:6790/0 conn(0x7fd69c02b4c0 legacy=0x7fd69c006890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:22:25.890 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.913+0000 7fd6a1ffb640 1 -- 172.21.15.47:0/1561224548 >> v1:172.21.15.121:6789/0 conn(0x7fd69c00b8a0 legacy=0x7fd69c0c0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:25.890 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.913+0000 7fd6a1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:22:25.890 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.913+0000 7fd6a1ffb640 1 monclient: found mon.noname-a 2024-03-29T11:22:25.890 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.913+0000 7fd6a1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:25.890 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.913+0000 7fd6a1ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-29T11:22:25.890 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.913+0000 7fd6a1ffb640 1 -- 172.21.15.47:0/1561224548 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd69c0cb160 con 0x7fd69c02a5a0 2024-03-29T11:22:25.890 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.913+0000 7fd6a1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:22:25.890 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.913+0000 7fd6a1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:22:25.890 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.913+0000 7fd6a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:25.890 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.913+0000 7fd6a3fff640 5 monclient: authenticate success, global_id 4554 2024-03-29T11:22:25.890 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.913+0000 7fd6a3fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:25.890 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.913+0000 7fd6a1ffb640 1 -- 172.21.15.47:0/1561224548 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fd698001690 con 0x7fd69c02a5a0 2024-03-29T11:22:25.890 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.913+0000 7fd6a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:25.891 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.913+0000 7fd6a1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:25.891 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.913+0000 7fd6a1ffb640 10 monclient: dump: 2024-03-29T11:22:25.891 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:22:25.891 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:22:25.891 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:25.891 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:25.891 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:25.891 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:22:25.891 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:22:25.891 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:22:25.891 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:22:25.891 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:22:25.891 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.913+0000 7fd6a3fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:25.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.914+0000 7fd6a1ffb640 1 -- 172.21.15.47:0/1561224548 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd6980018b0 con 0x7fd69c02a5a0 2024-03-29T11:22:25.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.914+0000 7fd6a1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:25.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.914+0000 7fd6a3fff640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:25.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.914+0000 7fd6a3fff640 4 set_mon_vals no callback set 2024-03-29T11:22:25.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.915+0000 7fd6a3fff640 10 monclient: shutdown 2024-03-29T11:22:25.893 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.915+0000 7fd6a3fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:25.893 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.915+0000 7fd6a3fff640 1 -- 172.21.15.47:0/1561224548 >> v1:172.21.15.47:6789/0 conn(0x7fd69c02a5a0 legacy=0x7fd69c0c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:25.893 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.915+0000 7fd6a3fff640 1 -- 172.21.15.47:0/1561224548 shutdown_connections 2024-03-29T11:22:25.893 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.915+0000 7fd6a3fff640 1 -- 172.21.15.47:0/1561224548 >> 172.21.15.47:0/1561224548 conn(0x7fd69c0058f0 msgr2=0x7fd69c005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:25.893 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.915+0000 7fd6a3fff640 1 -- 172.21.15.47:0/1561224548 shutdown_connections 2024-03-29T11:22:25.893 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.915+0000 7fd6a3fff640 1 -- 172.21.15.47:0/1561224548 wait complete. 2024-03-29T11:22:25.893 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.916+0000 7fd6a3fff640 10 monclient: build_initial_monmap 2024-03-29T11:22:25.893 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.916+0000 7fd6a3fff640 10 monclient: monmap: 2024-03-29T11:22:25.893 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:22:25.893 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:25.893 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:22:25.893 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:22:25.893 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:25.893 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:22:25.893 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:22:25.894 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:22:25.894 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:22:25.894 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:22:25.896 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.916+0000 7fd6a3fff640 1 librados: starting msgr at 2024-03-29T11:22:25.896 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.916+0000 7fd6a3fff640 1 librados: starting objecter 2024-03-29T11:22:25.896 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.917+0000 7fd6a3fff640 1 Processor -- start 2024-03-29T11:22:25.896 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.917+0000 7fd6a3fff640 1 -- start start 2024-03-29T11:22:25.896 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.917+0000 7fd6a3fff640 1 librados: setting wanted keys 2024-03-29T11:22:25.896 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.917+0000 7fd6a3fff640 1 librados: calling monclient init 2024-03-29T11:22:25.896 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.917+0000 7fd6a3fff640 10 monclient: init 2024-03-29T11:22:25.896 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.918+0000 7fd6a3fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:25.896 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.918+0000 7fd6a3fff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:22:25.896 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.918+0000 7fd6a3fff640 10 monclient(hunting): picked mon.noname-b con 0x7fd69c00b8a0 addr v1:172.21.15.121:6789/0 2024-03-29T11:22:25.896 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.918+0000 7fd6a3fff640 10 monclient(hunting): picked mon.noname-a con 0x7fd69c02b4c0 addr v1:172.21.15.47:6789/0 2024-03-29T11:22:25.896 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.918+0000 7fd6a3fff640 10 monclient(hunting): picked mon.noname-c con 0x7fd69c0c39a0 addr v1:172.21.15.47:6790/0 2024-03-29T11:22:25.896 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.918+0000 7fd6a3fff640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd69c0c48b0 con 0x7fd69c02b4c0 2024-03-29T11:22:25.897 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.918+0000 7fd6a3fff640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd69c0b5080 con 0x7fd69c00b8a0 2024-03-29T11:22:25.897 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.918+0000 7fd6a3fff640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd69c0b5360 con 0x7fd69c0c39a0 2024-03-29T11:22:25.897 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.918+0000 7fd6a3fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:25.897 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.918+0000 7fd6a3fff640 10 monclient(hunting): authenticate will time out at 1389.106204s 2024-03-29T11:22:25.901 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.924+0000 7fd6a37fe640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fd69c0c39a0 0x7fd69c0b0c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:35637/0 (socket says 172.21.15.47:35637) 2024-03-29T11:22:25.901 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.924+0000 7fd6a37fe640 1 -- 172.21.15.47:0/737701765 learned_addr learned my addr 172.21.15.47:0/737701765 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:22:25.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.927+0000 7fd68bfff640 10 client.?.objecter ms_handle_connect 0x7fd69c02b4c0 2024-03-29T11:22:25.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.927+0000 7fd68bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:25.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.927+0000 7fd68bfff640 10 client.?.objecter ms_handle_connect 0x7fd69c0c39a0 2024-03-29T11:22:25.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.927+0000 7fd68bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:25.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.927+0000 7fd68bfff640 10 client.?.objecter ms_handle_connect 0x7fd69c00b8a0 2024-03-29T11:22:25.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.927+0000 7fd68bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:25.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.927+0000 7fd68bfff640 1 -- 172.21.15.47:0/737701765 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 880903918 0 0) 0x7fd69c0b5360 con 0x7fd69c0c39a0 2024-03-29T11:22:25.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.927+0000 7fd68bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:25.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.927+0000 7fd68bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:25.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.927+0000 7fd68bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:25.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.927+0000 7fd68bfff640 10 monclient(hunting): my global_id is 4616 2024-03-29T11:22:25.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.927+0000 7fd68bfff640 1 -- 172.21.15.47:0/737701765 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd690004eb0 con 0x7fd69c0c39a0 2024-03-29T11:22:25.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.928+0000 7fd68bfff640 1 -- 172.21.15.47:0/737701765 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2785577879 0 0) 0x7fd69c0c48b0 con 0x7fd69c02b4c0 2024-03-29T11:22:25.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.928+0000 7fd68bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:25.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.928+0000 7fd68bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:25.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.928+0000 7fd68bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:25.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.928+0000 7fd68bfff640 10 monclient(hunting): my global_id is 4557 2024-03-29T11:22:25.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.928+0000 7fd68bfff640 1 -- 172.21.15.47:0/737701765 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd69c0b5360 con 0x7fd69c02b4c0 2024-03-29T11:22:25.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.929+0000 7fd68bfff640 1 -- 172.21.15.47:0/737701765 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2740911166 0 0) 0x7fd69c0b5080 con 0x7fd69c00b8a0 2024-03-29T11:22:25.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.929+0000 7fd68bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:25.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.929+0000 7fd68bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:25.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.929+0000 7fd68bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:25.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.931+0000 7fd68bfff640 10 monclient(hunting): my global_id is 4534 2024-03-29T11:22:25.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.931+0000 7fd68bfff640 1 -- 172.21.15.47:0/737701765 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd69c0c48b0 con 0x7fd69c00b8a0 2024-03-29T11:22:25.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.932+0000 7fd68bfff640 1 -- 172.21.15.47:0/737701765 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1479844648 0 0) 0x7fd690004eb0 con 0x7fd69c0c39a0 2024-03-29T11:22:25.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.932+0000 7fd68bfff640 1 -- 172.21.15.47:0/737701765 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd69c0b5080 con 0x7fd69c0c39a0 2024-03-29T11:22:25.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.932+0000 7fd68bfff640 1 -- 172.21.15.47:0/737701765 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 786036826 0 0) 0x7fd69c0b5360 con 0x7fd69c02b4c0 2024-03-29T11:22:25.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.933+0000 7fd68bfff640 1 -- 172.21.15.47:0/737701765 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd690004eb0 con 0x7fd69c02b4c0 2024-03-29T11:22:25.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.933+0000 7fd68bfff640 1 -- 172.21.15.47:0/737701765 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fd6980047e0 con 0x7fd69c0c39a0 2024-03-29T11:22:25.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.933+0000 7fd68bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:25.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.933+0000 7fd68bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:25.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.933+0000 7fd68bfff640 10 monclient(hunting): dump: 2024-03-29T11:22:25.911 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:22:25.911 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:22:25.912 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:25.912 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:25.912 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:25.912 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:22:25.912 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:22:25.912 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:22:25.912 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:22:25.912 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:22:25.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.933+0000 7fd68bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:25.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.933+0000 7fd68bfff640 1 -- 172.21.15.47:0/737701765 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fd6a404fdd0 con 0x7fd69c02b4c0 2024-03-29T11:22:25.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.933+0000 7fd68bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:25.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.933+0000 7fd68bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:25.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.933+0000 7fd68bfff640 10 monclient(hunting): dump: 2024-03-29T11:22:25.912 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:22:25.913 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:22:25.913 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:25.913 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:25.913 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:25.913 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:22:25.913 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:22:25.913 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:22:25.913 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:22:25.913 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:22:25.913 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.933+0000 7fd68bfff640 1 -- 172.21.15.47:0/737701765 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1667898413 0 0) 0x7fd69c0c48b0 con 0x7fd69c00b8a0 2024-03-29T11:22:25.913 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.934+0000 7fd68bfff640 1 -- 172.21.15.47:0/737701765 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd69c0b5360 con 0x7fd69c00b8a0 2024-03-29T11:22:25.913 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.934+0000 7fd68bfff640 1 -- 172.21.15.47:0/737701765 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fd694001760 con 0x7fd69c00b8a0 2024-03-29T11:22:25.913 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.934+0000 7fd68bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:25.913 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.934+0000 7fd68bfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:25.914 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.934+0000 7fd68bfff640 10 monclient(hunting): dump: 2024-03-29T11:22:25.914 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:22:25.914 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:22:25.914 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:25.914 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:25.914 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:25.914 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:22:25.914 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:22:25.914 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:22:25.914 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:22:25.914 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:22:25.914 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.934+0000 7fd68bfff640 1 -- 172.21.15.47:0/737701765 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3063877296 0 0) 0x7fd690004eb0 con 0x7fd69c02b4c0 2024-03-29T11:22:25.914 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.934+0000 7fd68bfff640 1 -- 172.21.15.47:0/737701765 >> v1:172.21.15.47:6790/0 conn(0x7fd69c0c39a0 legacy=0x7fd69c0b0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:25.918 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.934+0000 7fd68bfff640 1 -- 172.21.15.47:0/737701765 >> v1:172.21.15.121:6789/0 conn(0x7fd69c00b8a0 legacy=0x7fd69c0b5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:25.918 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.934+0000 7fd68bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:22:25.918 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.934+0000 7fd68bfff640 1 monclient: found mon.a 2024-03-29T11:22:25.918 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.934+0000 7fd68bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:25.918 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.934+0000 7fd68bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:22:25.918 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.934+0000 7fd68bfff640 1 -- 172.21.15.47:0/737701765 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd69c0c2ce0 con 0x7fd69c02b4c0 2024-03-29T11:22:25.918 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.934+0000 7fd68bfff640 10 monclient: _finish_auth 0 2024-03-29T11:22:25.918 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.934+0000 7fd68bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:22:25.918 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.934+0000 7fd68bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:25.918 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.934+0000 7fd6a3fff640 5 monclient: authenticate success, global_id 4557 2024-03-29T11:22:25.918 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.934+0000 7fd6a3fff640 10 monclient: _renew_subs 2024-03-29T11:22:25.918 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.934+0000 7fd6a3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:22:25.919 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.934+0000 7fd6a3fff640 1 -- 172.21.15.47:0/737701765 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd69c0c2ed0 con 0x7fd69c02b4c0 2024-03-29T11:22:25.919 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.935+0000 7fd6a3fff640 10 client.4557.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:25.919 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.935+0000 7fd6a3fff640 10 monclient: _renew_subs 2024-03-29T11:22:25.919 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.935+0000 7fd6a3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:22:25.919 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.935+0000 7fd6a3fff640 1 -- 172.21.15.47:0/737701765 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd69c0b3fd0 con 0x7fd69c02b4c0 2024-03-29T11:22:25.919 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.935+0000 7fd6a3fff640 1 librados: init done 2024-03-29T11:22:25.919 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.936+0000 7fd68bfff640 1 -- 172.21.15.47:0/737701765 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd6a404f400 con 0x7fd69c02b4c0 2024-03-29T11:22:25.919 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.936+0000 7fd68bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:25.919 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.936+0000 7fd6a17fa640 4 set_mon_vals no callback set 2024-03-29T11:22:25.919 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.937+0000 7fd68bfff640 1 -- 172.21.15.47:0/737701765 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fd6a4075710 con 0x7fd69c02b4c0 2024-03-29T11:22:25.919 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.937+0000 7fd68bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:25.919 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.937+0000 7fd68bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:25.919 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.937+0000 7fd68bfff640 10 monclient: dump: 2024-03-29T11:22:25.919 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:22:25.920 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:22:25.920 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:25.920 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:25.920 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:25.920 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:22:25.920 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:22:25.920 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:22:25.920 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:22:25.920 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:22:25.920 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.937+0000 7fd6a3fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:25.920 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.937+0000 7fd6a3fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:25.920 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.937+0000 7fd6a3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:22:25.920 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.937+0000 7fd6a3fff640 1 -- 172.21.15.47:0/737701765 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd684005080 con 0x7fd69c02b4c0 2024-03-29T11:22:25.920 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.941+0000 7fd68bfff640 1 -- 172.21.15.47:0/737701765 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7fd6a4067bf0 con 0x7fd69c02b4c0 2024-03-29T11:22:25.921 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.942+0000 7fd68bfff640 1 -- 172.21.15.47:0/737701765 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7fd6a40fe2b0 con 0x7fd69c02b4c0 2024-03-29T11:22:25.921 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.942+0000 7fd68bfff640 10 client.4557.objecter ms_dispatch 0x7fd6a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:25.922 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.942+0000 7fd68bfff640 3 client.4557.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:25.922 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.942+0000 7fd68bfff640 3 client.4557.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:25.922 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.942+0000 7fd68bfff640 20 client.4557.objecter dump_active .. 0 homeless 2024-03-29T11:22:25.922 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.943+0000 7fd6a2ffd640 10 monclient: get_auth_request con 0x7fd69006de60 auth_method 0 2024-03-29T11:22:25.922 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.944+0000 7fd68bfff640 10 client.4557.objecter ms_handle_connect 0x7fd69006de60 2024-03-29T11:22:25.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.950+0000 7fd68bfff640 1 -- 172.21.15.47:0/737701765 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd6a40c9b90 con 0x7fd69c02b4c0 2024-03-29T11:22:25.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.950+0000 7fd68bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:25.928 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:25.950+0000 7fd68bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:26.326 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:26.349+0000 7fd6a3fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:26.326 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:26.349+0000 7fd6a3fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:26.326 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:26.349+0000 7fd6a3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:22:26.326 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:26.349+0000 7fd6a3fff640 1 -- 172.21.15.47:0/737701765 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd684005330 con 0x7fd69c02b4c0 2024-03-29T11:22:26.328 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:26.351+0000 7fd68bfff640 1 -- 172.21.15.47:0/737701765 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7fd6a40cbac0 con 0x7fd69c02b4c0 2024-03-29T11:22:26.328 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:26.352+0000 7fd68bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:26.328 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:26.352+0000 7fd68bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:26.329 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:22:26.329 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:26.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:26.357+0000 7fd6a3fff640 10 librados: watch_flush enter 2024-03-29T11:22:26.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:26.357+0000 7fd6a3fff640 10 librados: watch_flush exit 2024-03-29T11:22:26.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:26.357+0000 7fd6a3fff640 20 client.4557.objecter shutdown clearing up homeless session... 2024-03-29T11:22:26.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:26.357+0000 7fd6a3fff640 10 client.4557.objecter successfully canceled tick 2024-03-29T11:22:26.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:26.358+0000 7fd6a3fff640 1 -- 172.21.15.47:0/737701765 >> v1:172.21.15.121:6816/1964117370 conn(0x7fd69006de60 legacy=0x7fd690070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:26.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:26.358+0000 7fd6a3fff640 10 monclient: shutdown 2024-03-29T11:22:26.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:26.358+0000 7fd6a3fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:26.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:26.358+0000 7fd6a3fff640 1 -- 172.21.15.47:0/737701765 >> v1:172.21.15.47:6789/0 conn(0x7fd69c02b4c0 legacy=0x7fd69c0c25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:26.336 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:26.358+0000 7fd6a3fff640 1 -- 172.21.15.47:0/737701765 shutdown_connections 2024-03-29T11:22:26.336 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:26.358+0000 7fd6a3fff640 1 -- 172.21.15.47:0/737701765 >> 172.21.15.47:0/737701765 conn(0x7fd69c0058f0 msgr2=0x7fd69c005cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:26.336 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:26.359+0000 7fd6a3fff640 1 -- 172.21.15.47:0/737701765 shutdown_connections 2024-03-29T11:22:26.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:26.359+0000 7fd6a3fff640 1 -- 172.21.15.47:0/737701765 wait complete. 2024-03-29T11:22:26.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:26.359+0000 7fd6a3fff640 1 librados: shutdown 2024-03-29T11:22:26.356 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:33.163 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:33.317 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.341+0000 7fa1f93d6640 10 monclient: get_monmap_and_config 2024-03-29T11:22:33.317 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.341+0000 7fa1f93d6640 10 monclient: build_initial_monmap 2024-03-29T11:22:33.317 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.341+0000 7fa1f93d6640 10 monclient: monmap: 2024-03-29T11:22:33.317 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:22:33.318 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:22:33.318 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:22:33.342021+0000 2024-03-29T11:22:33.318 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:22:33.342021+0000 2024-03-29T11:22:33.318 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:33.318 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:22:33.318 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:22:33.318 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:22:33.318 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:22:33.318 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:22:33.321 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.341+0000 7fa1f93d6640 1 Processor -- start 2024-03-29T11:22:33.321 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.342+0000 7fa1f93d6640 1 -- start start 2024-03-29T11:22:33.321 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.342+0000 7fa1f93d6640 10 monclient: init 2024-03-29T11:22:33.321 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.342+0000 7fa1f93d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:33.321 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.342+0000 7fa1f93d6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:22:33.321 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.342+0000 7fa1f93d6640 10 monclient(hunting): picked mon.noname-b con 0x7fa1f40b0770 addr v1:172.21.15.47:6790/0 2024-03-29T11:22:33.321 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.343+0000 7fa1f93d6640 10 monclient(hunting): picked mon.noname-a con 0x7fa1f414a700 addr v1:172.21.15.47:6789/0 2024-03-29T11:22:33.321 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.343+0000 7fa1f2ffd640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fa1f40b0770 0x7fa1f4149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:55968/0 (socket says 172.21.15.47:55968) 2024-03-29T11:22:33.321 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.343+0000 7fa1f93d6640 10 monclient(hunting): picked mon.noname-c con 0x7fa1f40ac5a0 addr v1:172.21.15.121:6789/0 2024-03-29T11:22:33.321 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.343+0000 7fa1f93d6640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1f40aba70 con 0x7fa1f414a700 2024-03-29T11:22:33.322 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.343+0000 7fa1f93d6640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1f40aecb0 con 0x7fa1f40ac5a0 2024-03-29T11:22:33.322 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.343+0000 7fa1f93d6640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1f40af150 con 0x7fa1f40b0770 2024-03-29T11:22:33.322 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.343+0000 7fa1f93d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:33.322 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.343+0000 7fa1f93d6640 10 monclient(hunting): authenticate will time out at 1396.531277s 2024-03-29T11:22:33.322 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.343+0000 7fa1f2ffd640 1 -- 172.21.15.47:0/3830841273 learned_addr learned my addr 172.21.15.47:0/3830841273 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:22:33.322 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.344+0000 7fa1f1ffb640 1 -- 172.21.15.47:0/3830841273 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3820044889 0 0) 0x7fa1f40af150 con 0x7fa1f40b0770 2024-03-29T11:22:33.322 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.344+0000 7fa1f1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:33.322 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.344+0000 7fa1f1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:33.322 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.344+0000 7fa1f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:33.322 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.344+0000 7fa1f1ffb640 10 monclient(hunting): my global_id is 4619 2024-03-29T11:22:33.322 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.344+0000 7fa1f1ffb640 1 -- 172.21.15.47:0/3830841273 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1dc003520 con 0x7fa1f40b0770 2024-03-29T11:22:33.322 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.344+0000 7fa1f1ffb640 1 -- 172.21.15.47:0/3830841273 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1308754976 0 0) 0x7fa1f40aba70 con 0x7fa1f414a700 2024-03-29T11:22:33.322 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.344+0000 7fa1f1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:33.323 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.344+0000 7fa1f1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:33.323 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.344+0000 7fa1f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:33.323 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.344+0000 7fa1f1ffb640 10 monclient(hunting): my global_id is 4560 2024-03-29T11:22:33.323 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.344+0000 7fa1f1ffb640 1 -- 172.21.15.47:0/3830841273 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1f40af150 con 0x7fa1f414a700 2024-03-29T11:22:33.323 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.344+0000 7fa1f1ffb640 1 -- 172.21.15.47:0/3830841273 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2560889223 0 0) 0x7fa1dc003520 con 0x7fa1f40b0770 2024-03-29T11:22:33.323 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.344+0000 7fa1f1ffb640 1 -- 172.21.15.47:0/3830841273 >> v1:172.21.15.121:6789/0 conn(0x7fa1f40ac5a0 legacy=0x7fa1f4151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:22:33.323 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.344+0000 7fa1f1ffb640 1 -- 172.21.15.47:0/3830841273 >> v1:172.21.15.47:6789/0 conn(0x7fa1f414a700 legacy=0x7fa1f40abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:33.323 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.344+0000 7fa1f1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:22:33.323 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.344+0000 7fa1f1ffb640 1 monclient: found mon.noname-b 2024-03-29T11:22:33.323 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.344+0000 7fa1f1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:33.323 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.344+0000 7fa1f1ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-29T11:22:33.323 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.344+0000 7fa1f1ffb640 1 -- 172.21.15.47:0/3830841273 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1f40ac9d0 con 0x7fa1f40b0770 2024-03-29T11:22:33.323 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.344+0000 7fa1f1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:22:33.323 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.345+0000 7fa1f1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:22:33.324 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.345+0000 7fa1f1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:33.324 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.345+0000 7fa1f1ffb640 1 -- 172.21.15.47:0/3830841273 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fa1e4001a10 con 0x7fa1f40b0770 2024-03-29T11:22:33.324 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.345+0000 7fa1f93d6640 5 monclient: authenticate success, global_id 4619 2024-03-29T11:22:33.324 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.345+0000 7fa1f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:33.324 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.345+0000 7fa1f1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:33.324 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.345+0000 7fa1f1ffb640 10 monclient: dump: 2024-03-29T11:22:33.324 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:22:33.324 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:22:33.324 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:33.324 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:33.324 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:33.324 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:22:33.324 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:22:33.324 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:22:33.324 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:22:33.324 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:22:33.325 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.345+0000 7fa1f93d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:33.332 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.345+0000 7fa1f1ffb640 1 -- 172.21.15.47:0/3830841273 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa1e4001c30 con 0x7fa1f40b0770 2024-03-29T11:22:33.332 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.345+0000 7fa1f1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:33.332 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.345+0000 7fa1f93d6640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:33.332 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.345+0000 7fa1f1ffb640 1 -- 172.21.15.47:0/3830841273 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fa1e4001fc0 con 0x7fa1f40b0770 2024-03-29T11:22:33.333 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.345+0000 7fa1f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:33.333 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.345+0000 7fa1f93d6640 4 set_mon_vals no callback set 2024-03-29T11:22:33.333 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.345+0000 7fa1f1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:33.333 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.345+0000 7fa1f1ffb640 10 monclient: dump: 2024-03-29T11:22:33.333 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:22:33.333 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:22:33.333 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:33.333 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:33.333 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:33.333 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:22:33.333 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:22:33.333 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:22:33.333 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:22:33.333 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:22:33.333 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.346+0000 7fa1f93d6640 10 monclient: shutdown 2024-03-29T11:22:33.333 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.346+0000 7fa1f93d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:33.334 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.346+0000 7fa1f93d6640 1 -- 172.21.15.47:0/3830841273 >> v1:172.21.15.47:6790/0 conn(0x7fa1f40b0770 legacy=0x7fa1f4149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:33.334 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.346+0000 7fa1f93d6640 1 -- 172.21.15.47:0/3830841273 shutdown_connections 2024-03-29T11:22:33.334 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.346+0000 7fa1f93d6640 1 -- 172.21.15.47:0/3830841273 >> 172.21.15.47:0/3830841273 conn(0x7fa1f406b660 msgr2=0x7fa1f406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:33.334 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.346+0000 7fa1f93d6640 1 -- 172.21.15.47:0/3830841273 shutdown_connections 2024-03-29T11:22:33.334 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.346+0000 7fa1f93d6640 1 -- 172.21.15.47:0/3830841273 wait complete. 2024-03-29T11:22:33.334 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.347+0000 7fa1f93d6640 10 monclient: build_initial_monmap 2024-03-29T11:22:33.334 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.347+0000 7fa1f93d6640 10 monclient: monmap: 2024-03-29T11:22:33.334 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:22:33.334 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:33.334 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:22:33.334 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:22:33.334 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:33.334 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:22:33.334 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:22:33.334 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:22:33.335 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:22:33.335 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:22:33.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.348+0000 7fa1f93d6640 1 librados: starting msgr at 2024-03-29T11:22:33.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.348+0000 7fa1f93d6640 1 librados: starting objecter 2024-03-29T11:22:33.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.348+0000 7fa1f93d6640 1 Processor -- start 2024-03-29T11:22:33.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.348+0000 7fa1f93d6640 1 -- start start 2024-03-29T11:22:33.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.348+0000 7fa1f93d6640 1 librados: setting wanted keys 2024-03-29T11:22:33.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.348+0000 7fa1f93d6640 1 librados: calling monclient init 2024-03-29T11:22:33.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.348+0000 7fa1f93d6640 10 monclient: init 2024-03-29T11:22:33.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.349+0000 7fa1f93d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:33.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.349+0000 7fa1f93d6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:22:33.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.349+0000 7fa1f93d6640 10 monclient(hunting): picked mon.noname-c con 0x7fa1f40ac5a0 addr v1:172.21.15.47:6790/0 2024-03-29T11:22:33.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.349+0000 7fa1f93d6640 10 monclient(hunting): picked mon.noname-b con 0x7fa1f40b0770 addr v1:172.21.15.121:6789/0 2024-03-29T11:22:33.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.349+0000 7fa1f93d6640 10 monclient(hunting): picked mon.noname-a con 0x7fa1f414a700 addr v1:172.21.15.47:6789/0 2024-03-29T11:22:33.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.349+0000 7fa1f93d6640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1f41282f0 con 0x7fa1f414a700 2024-03-29T11:22:33.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.349+0000 7fa1f93d6640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1f41db830 con 0x7fa1f40b0770 2024-03-29T11:22:33.336 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.349+0000 7fa1f93d6640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1f4125ed0 con 0x7fa1f40ac5a0 2024-03-29T11:22:33.336 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.349+0000 7fa1f93d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:33.336 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.349+0000 7fa1f93d6640 10 monclient(hunting): authenticate will time out at 1396.537448s 2024-03-29T11:22:33.336 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.349+0000 7fa1f2ffd640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fa1f40ac5a0 0x7fa1f41dbd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:57282/0 (socket says 172.21.15.47:57282) 2024-03-29T11:22:33.336 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.349+0000 7fa1f2ffd640 1 -- 172.21.15.47:0/683659299 learned_addr learned my addr 172.21.15.47:0/683659299 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:22:33.336 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.349+0000 7fa1db7fe640 10 client.?.objecter ms_handle_connect 0x7fa1f40ac5a0 2024-03-29T11:22:33.336 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.349+0000 7fa1db7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:33.336 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.350+0000 7fa1db7fe640 1 -- 172.21.15.47:0/683659299 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1229353291 0 0) 0x7fa1f4125ed0 con 0x7fa1f40ac5a0 2024-03-29T11:22:33.336 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.350+0000 7fa1db7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:22:33.336 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.350+0000 7fa1db7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:33.336 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.350+0000 7fa1db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:33.336 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.350+0000 7fa1db7fe640 10 monclient(hunting): my global_id is 4622 2024-03-29T11:22:33.336 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.350+0000 7fa1db7fe640 1 -- 172.21.15.47:0/683659299 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1e0004eb0 con 0x7fa1f40ac5a0 2024-03-29T11:22:33.336 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.350+0000 7fa1db7fe640 1 -- 172.21.15.47:0/683659299 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1554941829 0 0) 0x7fa1e0004eb0 con 0x7fa1f40ac5a0 2024-03-29T11:22:33.336 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.350+0000 7fa1db7fe640 1 -- 172.21.15.47:0/683659299 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa1f4125ed0 con 0x7fa1f40ac5a0 2024-03-29T11:22:33.336 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.350+0000 7fa1db7fe640 1 -- 172.21.15.47:0/683659299 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fa1e4001a10 con 0x7fa1f40ac5a0 2024-03-29T11:22:33.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.350+0000 7fa1db7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:33.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.350+0000 7fa1db7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:33.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.350+0000 7fa1db7fe640 10 monclient(hunting): dump: 2024-03-29T11:22:33.337 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:22:33.337 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:22:33.337 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:33.337 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:33.337 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:33.337 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:22:33.337 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:22:33.337 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:22:33.337 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:22:33.337 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:22:33.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.350+0000 7fa1db7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:33.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.351+0000 7fa1db7fe640 1 -- 172.21.15.47:0/683659299 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2410167607 0 0) 0x7fa1f4125ed0 con 0x7fa1f40ac5a0 2024-03-29T11:22:33.337 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.351+0000 7fa1db7fe640 1 -- 172.21.15.47:0/683659299 >> v1:172.21.15.121:6789/0 conn(0x7fa1f40b0770 legacy=0x7fa1f4144ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:33.338 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.351+0000 7fa1db7fe640 1 -- 172.21.15.47:0/683659299 >> v1:172.21.15.47:6789/0 conn(0x7fa1f414a700 legacy=0x7fa1f41e7090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:22:33.338 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.351+0000 7fa1db7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:22:33.338 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.351+0000 7fa1db7fe640 1 monclient: found mon.c 2024-03-29T11:22:33.338 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.351+0000 7fa1db7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:33.338 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.351+0000 7fa1db7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:22:33.338 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.351+0000 7fa1db7fe640 1 -- 172.21.15.47:0/683659299 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1f41239d0 con 0x7fa1f40ac5a0 2024-03-29T11:22:33.338 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.351+0000 7fa1db7fe640 10 monclient: _finish_auth 0 2024-03-29T11:22:33.338 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.351+0000 7fa1db7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:22:33.338 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.351+0000 7fa1db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:33.338 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.351+0000 7fa1f93d6640 5 monclient: authenticate success, global_id 4622 2024-03-29T11:22:33.338 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.351+0000 7fa1f93d6640 10 monclient: _renew_subs 2024-03-29T11:22:33.338 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.351+0000 7fa1f93d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:22:33.338 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.351+0000 7fa1f93d6640 1 -- 172.21.15.47:0/683659299 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1f41dc460 con 0x7fa1f40ac5a0 2024-03-29T11:22:33.339 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.352+0000 7fa1f93d6640 10 client.4622.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:33.339 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.352+0000 7fa1f93d6640 10 monclient: _renew_subs 2024-03-29T11:22:33.339 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.352+0000 7fa1f93d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:22:33.339 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.352+0000 7fa1f93d6640 1 -- 172.21.15.47:0/683659299 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1f41274a0 con 0x7fa1f40ac5a0 2024-03-29T11:22:33.339 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.352+0000 7fa1f93d6640 1 librados: init done 2024-03-29T11:22:33.339 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.352+0000 7fa1db7fe640 1 -- 172.21.15.47:0/683659299 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa1e4004e10 con 0x7fa1f40ac5a0 2024-03-29T11:22:33.339 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.352+0000 7fa1db7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:33.339 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.352+0000 7fa1db7fe640 1 -- 172.21.15.47:0/683659299 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fa1e4005220 con 0x7fa1f40ac5a0 2024-03-29T11:22:33.339 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.352+0000 7fa1db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:33.339 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.352+0000 7fa1db7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:33.339 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.352+0000 7fa1db7fe640 10 monclient: dump: 2024-03-29T11:22:33.339 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:22:33.339 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:22:33.339 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:33.339 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:33.340 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:33.340 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:22:33.340 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:22:33.340 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:22:33.340 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:22:33.340 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:22:33.340 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.352+0000 7fa1f17fa640 4 set_mon_vals no callback set 2024-03-29T11:22:33.340 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.354+0000 7fa1f93d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:33.340 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.354+0000 7fa1f93d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:33.340 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.354+0000 7fa1f93d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:22:33.340 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.354+0000 7fa1f93d6640 1 -- 172.21.15.47:0/683659299 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1d4005080 con 0x7fa1f40ac5a0 2024-03-29T11:22:33.340 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.355+0000 7fa1db7fe640 1 -- 172.21.15.47:0/683659299 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7fa1e4005380 con 0x7fa1f40ac5a0 2024-03-29T11:22:33.340 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.356+0000 7fa1db7fe640 1 -- 172.21.15.47:0/683659299 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7fa1e4084f90 con 0x7fa1f40ac5a0 2024-03-29T11:22:33.340 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.356+0000 7fa1db7fe640 10 client.4622.objecter ms_dispatch 0x7fa1f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:33.340 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.356+0000 7fa1db7fe640 3 client.4622.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:33.341 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.356+0000 7fa1db7fe640 3 client.4622.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:33.341 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.356+0000 7fa1db7fe640 20 client.4622.objecter dump_active .. 0 homeless 2024-03-29T11:22:33.341 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.357+0000 7fa1f27fc640 10 monclient: get_auth_request con 0x7fa1e006da40 auth_method 0 2024-03-29T11:22:33.341 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.357+0000 7fa1db7fe640 10 client.4622.objecter ms_handle_connect 0x7fa1e006da40 2024-03-29T11:22:33.341 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.363+0000 7fa1db7fe640 1 -- 172.21.15.47:0/683659299 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa1e40b28b0 con 0x7fa1f40ac5a0 2024-03-29T11:22:33.342 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.363+0000 7fa1db7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:33.342 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.363+0000 7fa1db7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:33.656 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.680+0000 7fa1f93d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:33.746 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.680+0000 7fa1f93d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:33.746 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.680+0000 7fa1f93d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:22:33.746 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.680+0000 7fa1f93d6640 1 -- 172.21.15.47:0/683659299 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa1d4005330 con 0x7fa1f40ac5a0 2024-03-29T11:22:33.747 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.681+0000 7fa1db7fe640 1 -- 172.21.15.47:0/683659299 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7fa1e40506a0 con 0x7fa1f40ac5a0 2024-03-29T11:22:33.747 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.681+0000 7fa1db7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:33.747 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.681+0000 7fa1db7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:33.747 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:22:33.747 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:33.749 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.685+0000 7fa1f93d6640 10 librados: watch_flush enter 2024-03-29T11:22:33.749 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.685+0000 7fa1f93d6640 10 librados: watch_flush exit 2024-03-29T11:22:33.749 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.685+0000 7fa1f93d6640 20 client.4622.objecter shutdown clearing up homeless session... 2024-03-29T11:22:33.749 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.685+0000 7fa1f93d6640 10 client.4622.objecter successfully canceled tick 2024-03-29T11:22:33.749 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.686+0000 7fa1f93d6640 1 -- 172.21.15.47:0/683659299 >> v1:172.21.15.121:6816/1964117370 conn(0x7fa1e006da40 legacy=0x7fa1e006ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:33.749 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.686+0000 7fa1f93d6640 10 monclient: shutdown 2024-03-29T11:22:33.749 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.686+0000 7fa1f93d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:33.749 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.686+0000 7fa1f93d6640 1 -- 172.21.15.47:0/683659299 >> v1:172.21.15.47:6790/0 conn(0x7fa1f40ac5a0 legacy=0x7fa1f41dbd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:33.749 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.686+0000 7fa1f93d6640 1 -- 172.21.15.47:0/683659299 shutdown_connections 2024-03-29T11:22:33.749 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.686+0000 7fa1f93d6640 1 -- 172.21.15.47:0/683659299 >> 172.21.15.47:0/683659299 conn(0x7fa1f406b660 msgr2=0x7fa1f40b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:33.749 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.686+0000 7fa1f93d6640 1 -- 172.21.15.47:0/683659299 shutdown_connections 2024-03-29T11:22:33.749 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.686+0000 7fa1f93d6640 1 -- 172.21.15.47:0/683659299 wait complete. 2024-03-29T11:22:33.750 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:33.686+0000 7fa1f93d6640 1 librados: shutdown 2024-03-29T11:22:33.750 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:40.557 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:40.707 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.730+0000 7fd50fca8640 10 monclient: get_monmap_and_config 2024-03-29T11:22:40.707 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.730+0000 7fd50fca8640 10 monclient: build_initial_monmap 2024-03-29T11:22:40.707 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.730+0000 7fd50fca8640 10 monclient: monmap: 2024-03-29T11:22:40.707 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:22:40.707 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:22:40.707 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:22:40.731344+0000 2024-03-29T11:22:40.707 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:22:40.731344+0000 2024-03-29T11:22:40.707 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:40.707 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:22:40.707 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:22:40.708 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:22:40.708 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:22:40.708 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:22:40.709 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.731+0000 7fd50fca8640 1 Processor -- start 2024-03-29T11:22:40.709 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.731+0000 7fd50fca8640 1 -- start start 2024-03-29T11:22:40.709 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.731+0000 7fd50fca8640 10 monclient: init 2024-03-29T11:22:40.709 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.732+0000 7fd50fca8640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:40.709 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.732+0000 7fd50fca8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:22:40.709 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.732+0000 7fd50fca8640 10 monclient(hunting): picked mon.noname-a con 0x7fd5081457f0 addr v1:172.21.15.47:6789/0 2024-03-29T11:22:40.710 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.732+0000 7fd50fca8640 10 monclient(hunting): picked mon.noname-c con 0x7fd50813f650 addr v1:172.21.15.121:6789/0 2024-03-29T11:22:40.710 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.732+0000 7fd50fca8640 10 monclient(hunting): picked mon.noname-b con 0x7fd5080a9d70 addr v1:172.21.15.47:6790/0 2024-03-29T11:22:40.710 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.732+0000 7fd50fca8640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5080aaef0 con 0x7fd5081457f0 2024-03-29T11:22:40.711 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.732+0000 7fd50fca8640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd508145bd0 con 0x7fd50813f650 2024-03-29T11:22:40.711 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.732+0000 7fd50fca8640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd508145eb0 con 0x7fd5080a9d70 2024-03-29T11:22:40.711 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.732+0000 7fd50fca8640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:40.711 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.732+0000 7fd50fca8640 10 monclient(hunting): authenticate will time out at 1403.920582s 2024-03-29T11:22:40.711 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.734+0000 7fd50d243640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fd50813f650 0x7fd5080b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.47:54182/0 (socket says 172.21.15.47:54182) 2024-03-29T11:22:40.711 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.734+0000 7fd50d243640 1 -- 172.21.15.47:0/109792975 learned_addr learned my addr 172.21.15.47:0/109792975 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:22:40.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.735+0000 7fd50ca42640 1 -- 172.21.15.47:0/109792975 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3356429885 0 0) 0x7fd508145bd0 con 0x7fd50813f650 2024-03-29T11:22:40.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.735+0000 7fd50ca42640 10 monclient(hunting): _negotiate 2024-03-29T11:22:40.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.735+0000 7fd50ca42640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:40.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.735+0000 7fd50ca42640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:40.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.735+0000 7fd50ca42640 10 monclient(hunting): my global_id is 4537 2024-03-29T11:22:40.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.735+0000 7fd50ca42640 1 -- 172.21.15.47:0/109792975 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4f4003520 con 0x7fd50813f650 2024-03-29T11:22:40.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.736+0000 7fd50ca42640 1 -- 172.21.15.47:0/109792975 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3533200919 0 0) 0x7fd508145eb0 con 0x7fd5080a9d70 2024-03-29T11:22:40.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.736+0000 7fd50ca42640 10 monclient(hunting): _negotiate 2024-03-29T11:22:40.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.736+0000 7fd50ca42640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:40.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.736+0000 7fd50ca42640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:40.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.736+0000 7fd50ca42640 10 monclient(hunting): my global_id is 4625 2024-03-29T11:22:40.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.736+0000 7fd50ca42640 1 -- 172.21.15.47:0/109792975 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd508145bd0 con 0x7fd5080a9d70 2024-03-29T11:22:40.715 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.736+0000 7fd50ca42640 1 -- 172.21.15.47:0/109792975 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2725114252 0 0) 0x7fd508145bd0 con 0x7fd5080a9d70 2024-03-29T11:22:40.715 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.736+0000 7fd50ca42640 1 -- 172.21.15.47:0/109792975 >> v1:172.21.15.121:6789/0 conn(0x7fd50813f650 legacy=0x7fd5080b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:40.715 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.736+0000 7fd50ca42640 1 -- 172.21.15.47:0/109792975 >> v1:172.21.15.47:6789/0 conn(0x7fd5081457f0 legacy=0x7fd50813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:22:40.715 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.736+0000 7fd50ca42640 10 monclient: _finish_hunting 0 2024-03-29T11:22:40.715 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.736+0000 7fd50ca42640 1 monclient: found mon.noname-b 2024-03-29T11:22:40.715 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.736+0000 7fd50ca42640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:40.715 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.736+0000 7fd50ca42640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-29T11:22:40.715 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.736+0000 7fd50ca42640 1 -- 172.21.15.47:0/109792975 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd508146340 con 0x7fd5080a9d70 2024-03-29T11:22:40.715 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.736+0000 7fd50ca42640 10 monclient: _finish_auth 0 2024-03-29T11:22:40.715 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.736+0000 7fd50ca42640 10 monclient: _check_auth_tickets 2024-03-29T11:22:40.715 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.736+0000 7fd50ca42640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:40.715 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.736+0000 7fd50ca42640 1 -- 172.21.15.47:0/109792975 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fd5000019f0 con 0x7fd5080a9d70 2024-03-29T11:22:40.715 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.736+0000 7fd50ca42640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:40.715 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.736+0000 7fd50ca42640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:40.715 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.736+0000 7fd50ca42640 10 monclient: dump: 2024-03-29T11:22:40.716 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:22:40.716 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:22:40.716 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:40.716 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:40.716 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:40.716 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:22:40.716 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:22:40.716 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:22:40.716 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:22:40.716 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:22:40.716 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.736+0000 7fd50fca8640 5 monclient: authenticate success, global_id 4625 2024-03-29T11:22:40.716 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.736+0000 7fd50fca8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:40.718 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.738+0000 7fd50ca42640 1 -- 172.21.15.47:0/109792975 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd500001b50 con 0x7fd5080a9d70 2024-03-29T11:22:40.718 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.738+0000 7fd50ca42640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:40.718 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.738+0000 7fd50ca42640 1 -- 172.21.15.47:0/109792975 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fd500001fa0 con 0x7fd5080a9d70 2024-03-29T11:22:40.718 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.738+0000 7fd50ca42640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:40.719 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.738+0000 7fd50ca42640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:40.719 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.738+0000 7fd50ca42640 10 monclient: dump: 2024-03-29T11:22:40.719 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:22:40.719 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:22:40.719 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:40.719 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:40.719 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:40.719 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:22:40.719 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:22:40.719 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:22:40.719 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:22:40.719 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:22:40.719 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.738+0000 7fd50fca8640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:40.719 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.738+0000 7fd50fca8640 4 set_mon_vals no callback set 2024-03-29T11:22:40.719 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.739+0000 7fd50fca8640 10 monclient: shutdown 2024-03-29T11:22:40.720 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.739+0000 7fd50fca8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:40.720 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.739+0000 7fd50fca8640 1 -- 172.21.15.47:0/109792975 >> v1:172.21.15.47:6790/0 conn(0x7fd5080a9d70 legacy=0x7fd50814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:40.720 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.739+0000 7fd50fca8640 1 -- 172.21.15.47:0/109792975 shutdown_connections 2024-03-29T11:22:40.720 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.739+0000 7fd50fca8640 1 -- 172.21.15.47:0/109792975 >> 172.21.15.47:0/109792975 conn(0x7fd508066a00 msgr2=0x7fd508066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:40.720 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.740+0000 7fd50fca8640 1 -- 172.21.15.47:0/109792975 shutdown_connections 2024-03-29T11:22:40.720 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.740+0000 7fd50fca8640 1 -- 172.21.15.47:0/109792975 wait complete. 2024-03-29T11:22:40.720 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.741+0000 7fd50fca8640 10 monclient: build_initial_monmap 2024-03-29T11:22:40.722 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.741+0000 7fd50fca8640 10 monclient: monmap: 2024-03-29T11:22:40.722 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:22:40.722 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:40.722 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:22:40.722 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:22:40.722 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:40.722 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:22:40.722 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:22:40.723 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:22:40.723 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:22:40.723 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:22:40.724 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.742+0000 7fd50fca8640 1 librados: starting msgr at 2024-03-29T11:22:40.725 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.742+0000 7fd50fca8640 1 librados: starting objecter 2024-03-29T11:22:40.725 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.742+0000 7fd50fca8640 1 Processor -- start 2024-03-29T11:22:40.725 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.742+0000 7fd50fca8640 1 -- start start 2024-03-29T11:22:40.725 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.742+0000 7fd50fca8640 1 librados: setting wanted keys 2024-03-29T11:22:40.725 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.742+0000 7fd50fca8640 1 librados: calling monclient init 2024-03-29T11:22:40.725 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.742+0000 7fd50fca8640 10 monclient: init 2024-03-29T11:22:40.725 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.743+0000 7fd50fca8640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:40.725 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.743+0000 7fd50fca8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:22:40.725 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.743+0000 7fd50fca8640 10 monclient(hunting): picked mon.noname-a con 0x7fd5080a9d70 addr v1:172.21.15.47:6789/0 2024-03-29T11:22:40.725 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.743+0000 7fd50fca8640 10 monclient(hunting): picked mon.noname-b con 0x7fd50813f650 addr v1:172.21.15.121:6789/0 2024-03-29T11:22:40.725 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.743+0000 7fd50fca8640 10 monclient(hunting): picked mon.noname-c con 0x7fd5081457f0 addr v1:172.21.15.47:6790/0 2024-03-29T11:22:40.725 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.743+0000 7fd50fca8640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd508070ca0 con 0x7fd5080a9d70 2024-03-29T11:22:40.726 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.745+0000 7fd50fca8640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd508070f80 con 0x7fd50813f650 2024-03-29T11:22:40.726 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.745+0000 7fd50fca8640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd50806fee0 con 0x7fd5081457f0 2024-03-29T11:22:40.726 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.745+0000 7fd50fca8640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:40.726 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.745+0000 7fd50fca8640 10 monclient(hunting): authenticate will time out at 1403.932990s 2024-03-29T11:22:40.726 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.745+0000 7fd50d243640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fd50813f650 0x7fd5080713a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.47:54452/0 (socket says 172.21.15.47:54452) 2024-03-29T11:22:40.726 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.745+0000 7fd50d243640 1 -- 172.21.15.47:0/964603121 learned_addr learned my addr 172.21.15.47:0/964603121 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:22:40.726 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.747+0000 7fd4fe7fc640 10 client.?.objecter ms_handle_connect 0x7fd50813f650 2024-03-29T11:22:40.726 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.747+0000 7fd4fe7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:40.726 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.747+0000 7fd4fe7fc640 10 client.?.objecter ms_handle_connect 0x7fd5081457f0 2024-03-29T11:22:40.726 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.747+0000 7fd4fe7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:40.726 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.747+0000 7fd4fe7fc640 1 -- 172.21.15.47:0/964603121 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 557077700 0 0) 0x7fd50806fee0 con 0x7fd5081457f0 2024-03-29T11:22:40.726 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.747+0000 7fd4fe7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:40.726 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.747+0000 7fd4fe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:40.726 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.747+0000 7fd4fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:40.727 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.747+0000 7fd4fe7fc640 10 monclient(hunting): my global_id is 4628 2024-03-29T11:22:40.727 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.747+0000 7fd4fe7fc640 1 -- 172.21.15.47:0/964603121 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4ec004eb0 con 0x7fd5081457f0 2024-03-29T11:22:40.733 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.748+0000 7fd4fe7fc640 10 client.?.objecter ms_handle_connect 0x7fd5080a9d70 2024-03-29T11:22:40.733 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.748+0000 7fd4fe7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:40.734 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.748+0000 7fd4fe7fc640 1 -- 172.21.15.47:0/964603121 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2558840341 0 0) 0x7fd508070f80 con 0x7fd50813f650 2024-03-29T11:22:40.734 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.748+0000 7fd4fe7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:40.734 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.748+0000 7fd4fe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:40.734 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.748+0000 7fd4fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:40.734 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.748+0000 7fd4fe7fc640 10 monclient(hunting): my global_id is 4540 2024-03-29T11:22:40.734 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.748+0000 7fd4fe7fc640 1 -- 172.21.15.47:0/964603121 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd50806fee0 con 0x7fd50813f650 2024-03-29T11:22:40.734 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.749+0000 7fd4fe7fc640 1 -- 172.21.15.47:0/964603121 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2552498916 0 0) 0x7fd508070ca0 con 0x7fd5080a9d70 2024-03-29T11:22:40.734 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.749+0000 7fd4fe7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:40.734 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.749+0000 7fd4fe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:40.734 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.749+0000 7fd4fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:40.734 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.749+0000 7fd4fe7fc640 10 monclient(hunting): my global_id is 4563 2024-03-29T11:22:40.734 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.749+0000 7fd4fe7fc640 1 -- 172.21.15.47:0/964603121 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd508070f80 con 0x7fd5080a9d70 2024-03-29T11:22:40.734 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.749+0000 7fd4fe7fc640 1 -- 172.21.15.47:0/964603121 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3763530137 0 0) 0x7fd4ec004eb0 con 0x7fd5081457f0 2024-03-29T11:22:40.735 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.749+0000 7fd4fe7fc640 1 -- 172.21.15.47:0/964603121 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd508070ca0 con 0x7fd5081457f0 2024-03-29T11:22:40.735 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.749+0000 7fd4fe7fc640 1 -- 172.21.15.47:0/964603121 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fd500001f20 con 0x7fd5081457f0 2024-03-29T11:22:40.735 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.749+0000 7fd4fe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:40.735 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.749+0000 7fd4fe7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:40.735 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.749+0000 7fd4fe7fc640 10 monclient(hunting): dump: 2024-03-29T11:22:40.735 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:22:40.735 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:22:40.735 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:40.735 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:40.735 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:40.735 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:22:40.735 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:22:40.735 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:22:40.735 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:22:40.735 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:22:40.736 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.749+0000 7fd4fe7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:40.736 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.749+0000 7fd4fe7fc640 1 -- 172.21.15.47:0/964603121 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 294118220 0 0) 0x7fd50806fee0 con 0x7fd50813f650 2024-03-29T11:22:40.736 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.749+0000 7fd4fe7fc640 1 -- 172.21.15.47:0/964603121 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd4ec004eb0 con 0x7fd50813f650 2024-03-29T11:22:40.736 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.749+0000 7fd4fe7fc640 1 -- 172.21.15.47:0/964603121 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fd4f80018d0 con 0x7fd50813f650 2024-03-29T11:22:40.736 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.749+0000 7fd4fe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:40.736 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.749+0000 7fd4fe7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:40.736 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.749+0000 7fd4fe7fc640 10 monclient(hunting): dump: 2024-03-29T11:22:40.736 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:22:40.736 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:22:40.736 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:40.736 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:40.736 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:40.736 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:22:40.737 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:22:40.737 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:22:40.737 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:22:40.737 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:22:40.737 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.749+0000 7fd4fe7fc640 1 -- 172.21.15.47:0/964603121 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 170153039 0 0) 0x7fd508070f80 con 0x7fd5080a9d70 2024-03-29T11:22:40.737 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.749+0000 7fd4fe7fc640 1 -- 172.21.15.47:0/964603121 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd50806fee0 con 0x7fd5080a9d70 2024-03-29T11:22:40.737 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.749+0000 7fd4fe7fc640 1 -- 172.21.15.47:0/964603121 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fd504001730 con 0x7fd5080a9d70 2024-03-29T11:22:40.737 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.749+0000 7fd4fe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:40.737 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.749+0000 7fd4fe7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:40.737 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.749+0000 7fd4fe7fc640 10 monclient(hunting): dump: 2024-03-29T11:22:40.737 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:22:40.737 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:22:40.737 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:40.737 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:40.738 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:40.738 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:22:40.738 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:22:40.738 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:22:40.738 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:22:40.738 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:22:40.738 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.749+0000 7fd4fe7fc640 1 -- 172.21.15.47:0/964603121 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2910963189 0 0) 0x7fd508070ca0 con 0x7fd5081457f0 2024-03-29T11:22:40.738 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.749+0000 7fd4fe7fc640 1 -- 172.21.15.47:0/964603121 >> v1:172.21.15.121:6789/0 conn(0x7fd50813f650 legacy=0x7fd5080713a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:40.738 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.749+0000 7fd4fe7fc640 1 -- 172.21.15.47:0/964603121 >> v1:172.21.15.47:6789/0 conn(0x7fd5080a9d70 legacy=0x7fd50806ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:40.738 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.749+0000 7fd4fe7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:22:40.738 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.749+0000 7fd4fe7fc640 1 monclient: found mon.c 2024-03-29T11:22:40.738 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.749+0000 7fd4fe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:40.738 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.749+0000 7fd4fe7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:22:40.739 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.749+0000 7fd4fe7fc640 1 -- 172.21.15.47:0/964603121 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5081393b0 con 0x7fd5081457f0 2024-03-29T11:22:40.739 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.750+0000 7fd4fe7fc640 10 monclient: _finish_auth 0 2024-03-29T11:22:40.739 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.750+0000 7fd4fe7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:22:40.739 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.750+0000 7fd4fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:40.739 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.750+0000 7fd50fca8640 5 monclient: authenticate success, global_id 4628 2024-03-29T11:22:40.739 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.750+0000 7fd50fca8640 10 monclient: _renew_subs 2024-03-29T11:22:40.739 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.750+0000 7fd50fca8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:22:40.739 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.750+0000 7fd50fca8640 1 -- 172.21.15.47:0/964603121 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5081e6f50 con 0x7fd5081457f0 2024-03-29T11:22:40.739 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.750+0000 7fd50fca8640 10 client.4628.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:40.739 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.750+0000 7fd50fca8640 10 monclient: _renew_subs 2024-03-29T11:22:40.739 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.750+0000 7fd50fca8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:22:40.739 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.750+0000 7fd50fca8640 1 -- 172.21.15.47:0/964603121 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5081e72c0 con 0x7fd5081457f0 2024-03-29T11:22:40.739 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.750+0000 7fd50fca8640 1 librados: init done 2024-03-29T11:22:40.740 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.750+0000 7fd4fe7fc640 1 -- 172.21.15.47:0/964603121 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd500003c20 con 0x7fd5081457f0 2024-03-29T11:22:40.740 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.750+0000 7fd4fe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:40.740 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.750+0000 7fd4fe7fc640 1 -- 172.21.15.47:0/964603121 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fd500003fb0 con 0x7fd5081457f0 2024-03-29T11:22:40.740 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.750+0000 7fd4fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:40.740 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.750+0000 7fd4fe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:40.740 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.750+0000 7fd4fe7fc640 10 monclient: dump: 2024-03-29T11:22:40.740 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:22:40.740 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:22:40.740 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:40.740 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:40.740 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:40.740 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:22:40.740 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:22:40.740 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:22:40.741 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:22:40.741 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:22:40.741 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.750+0000 7fd4fffff640 4 set_mon_vals no callback set 2024-03-29T11:22:40.741 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.751+0000 7fd50fca8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:40.741 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.751+0000 7fd50fca8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:40.741 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.751+0000 7fd50fca8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:22:40.741 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.751+0000 7fd50fca8640 1 -- 172.21.15.47:0/964603121 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4f0005080 con 0x7fd5081457f0 2024-03-29T11:22:40.741 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.753+0000 7fd4fe7fc640 1 -- 172.21.15.47:0/964603121 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7fd500019f80 con 0x7fd5081457f0 2024-03-29T11:22:40.741 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.754+0000 7fd4fe7fc640 1 -- 172.21.15.47:0/964603121 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1709489940 0 0) 0x7fd500083bb0 con 0x7fd5081457f0 2024-03-29T11:22:40.741 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.754+0000 7fd4fe7fc640 10 client.4628.objecter ms_dispatch 0x7fd508000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:40.741 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.754+0000 7fd4fe7fc640 3 client.4628.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:40.741 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.754+0000 7fd4fe7fc640 3 client.4628.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:40.741 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.754+0000 7fd4fe7fc640 20 client.4628.objecter dump_active .. 0 homeless 2024-03-29T11:22:40.741 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.756+0000 7fd50da44640 10 monclient: get_auth_request con 0x7fd4ec06e120 auth_method 0 2024-03-29T11:22:40.741 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.756+0000 7fd4fe7fc640 10 client.4628.objecter ms_handle_connect 0x7fd4ec06e120 2024-03-29T11:22:40.742 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.761+0000 7fd4fe7fc640 1 -- 172.21.15.47:0/964603121 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd50004f490 con 0x7fd5081457f0 2024-03-29T11:22:40.742 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.761+0000 7fd4fe7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:40.742 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:40.761+0000 7fd4fe7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:41.087 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:41.109+0000 7fd50fca8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:41.087 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:41.110+0000 7fd50fca8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:41.087 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:41.110+0000 7fd50fca8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:22:41.087 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:41.110+0000 7fd50fca8640 1 -- 172.21.15.47:0/964603121 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd4f0005330 con 0x7fd5081457f0 2024-03-29T11:22:41.092 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:41.115+0000 7fd4fe7fc640 1 -- 172.21.15.47:0/964603121 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3839215665) 0x7fd5000513c0 con 0x7fd5081457f0 2024-03-29T11:22:41.092 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:41.115+0000 7fd4fe7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:41.092 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:41.115+0000 7fd4fe7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:41.093 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:22:41.093 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:12:43.246801+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:12:43.246801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:41.095 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:41.118+0000 7fd50fca8640 10 librados: watch_flush enter 2024-03-29T11:22:41.095 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:41.118+0000 7fd50fca8640 10 librados: watch_flush exit 2024-03-29T11:22:41.095 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:41.118+0000 7fd50fca8640 20 client.4628.objecter shutdown clearing up homeless session... 2024-03-29T11:22:41.096 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:41.119+0000 7fd50fca8640 10 client.4628.objecter successfully canceled tick 2024-03-29T11:22:41.096 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:41.119+0000 7fd50fca8640 1 -- 172.21.15.47:0/964603121 >> v1:172.21.15.121:6816/1964117370 conn(0x7fd4ec06e120 legacy=0x7fd4ec0705e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:41.096 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:41.119+0000 7fd50fca8640 10 monclient: shutdown 2024-03-29T11:22:41.096 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:41.119+0000 7fd50fca8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:41.096 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:41.119+0000 7fd50fca8640 1 -- 172.21.15.47:0/964603121 >> v1:172.21.15.47:6790/0 conn(0x7fd5081457f0 legacy=0x7fd50806f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:41.097 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:41.119+0000 7fd50fca8640 1 -- 172.21.15.47:0/964603121 shutdown_connections 2024-03-29T11:22:41.097 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:41.119+0000 7fd50fca8640 1 -- 172.21.15.47:0/964603121 >> 172.21.15.47:0/964603121 conn(0x7fd508066a00 msgr2=0x7fd508066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:41.097 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:41.120+0000 7fd50fca8640 1 -- 172.21.15.47:0/964603121 shutdown_connections 2024-03-29T11:22:41.097 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:41.120+0000 7fd50fca8640 1 -- 172.21.15.47:0/964603121 wait complete. 2024-03-29T11:22:41.097 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:41.120+0000 7fd50fca8640 1 librados: shutdown 2024-03-29T11:22:41.113 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:47.921 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:48.065 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.086+0000 7f097daf4640 10 monclient: get_monmap_and_config 2024-03-29T11:22:48.065 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.086+0000 7f097daf4640 10 monclient: build_initial_monmap 2024-03-29T11:22:48.065 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.086+0000 7f097daf4640 10 monclient: monmap: 2024-03-29T11:22:48.065 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:22:48.066 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:22:48.066 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:22:48.087891+0000 2024-03-29T11:22:48.066 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:22:48.087891+0000 2024-03-29T11:22:48.066 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:48.066 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:22:48.066 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:22:48.066 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:22:48.066 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:22:48.066 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:22:48.066 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.088+0000 7f097daf4640 1 Processor -- start 2024-03-29T11:22:48.066 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.088+0000 7f097daf4640 1 -- start start 2024-03-29T11:22:48.066 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.088+0000 7f097daf4640 10 monclient: init 2024-03-29T11:22:48.067 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.089+0000 7f097daf4640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:48.067 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.089+0000 7f097daf4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:22:48.067 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.089+0000 7f097daf4640 10 monclient(hunting): picked mon.noname-c con 0x7f09780acf60 addr v1:172.21.15.121:6789/0 2024-03-29T11:22:48.067 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.089+0000 7f097daf4640 10 monclient(hunting): picked mon.noname-a con 0x7f09780ad5c0 addr v1:172.21.15.47:6789/0 2024-03-29T11:22:48.067 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.089+0000 7f097daf4640 10 monclient(hunting): picked mon.noname-b con 0x7f097813a000 addr v1:172.21.15.47:6790/0 2024-03-29T11:22:48.067 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.089+0000 7f097daf4640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09780aec00 con 0x7f09780ad5c0 2024-03-29T11:22:48.068 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.089+0000 7f097daf4640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f097813daa0 con 0x7f09780acf60 2024-03-29T11:22:48.068 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.089+0000 7f097daf4640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09780ad9d0 con 0x7f097813a000 2024-03-29T11:22:48.068 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.089+0000 7f097daf4640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:48.068 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.089+0000 7f097daf4640 10 monclient(hunting): authenticate will time out at 1411.277697s 2024-03-29T11:22:48.068 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.090+0000 7f0977fff640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f097813a000 0x7f097814b880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:39472/0 (socket says 172.21.15.47:39472) 2024-03-29T11:22:48.068 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.090+0000 7f0977fff640 1 -- 172.21.15.47:0/3676270523 learned_addr learned my addr 172.21.15.47:0/3676270523 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:22:48.068 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.090+0000 7f0977fff640 1 -- 172.21.15.47:0/3676270523 >> v1:172.21.15.47:6790/0 conn(0x7f097813a000 legacy=0x7f097814b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-29T11:22:48.068 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.090+0000 7f0977fff640 1 -- 172.21.15.47:0/3676270523 >> v1:172.21.15.47:6790/0 conn(0x7f097813a000 legacy=0x7f097814b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:22:48.068 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.090+0000 7f0977fff640 1 --1- 172.21.15.47:0/3676270523 >> v1:172.21.15.47:6790/0 conn(0x7f097813a000 0x7f097814b880 :-1 s=OPENED pgs=1005552 cs=1 l=1).handle_message read tag failed 2024-03-29T11:22:48.068 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.090+0000 7f0977fff640 1 --1- 172.21.15.47:0/3676270523 >> v1:172.21.15.47:6790/0 conn(0x7f097813a000 0x7f097814b880 :-1 s=OPENED pgs=1005552 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:22:48.068 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.090+0000 7f09767fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.47:6790/0 2024-03-29T11:22:48.070 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.093+0000 7f09767fc640 1 -- 172.21.15.47:0/3676270523 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1007595935 0 0) 0x7f09780aec00 con 0x7f09780ad5c0 2024-03-29T11:22:48.070 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.093+0000 7f09767fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:48.070 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.093+0000 7f09767fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:48.070 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.093+0000 7f09767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:48.071 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.093+0000 7f09767fc640 10 monclient(hunting): my global_id is 4566 2024-03-29T11:22:48.071 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.093+0000 7f09767fc640 1 -- 172.21.15.47:0/3676270523 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0960003520 con 0x7f09780ad5c0 2024-03-29T11:22:48.071 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.094+0000 7f09767fc640 1 -- 172.21.15.47:0/3676270523 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2161530728 0 0) 0x7f0960003520 con 0x7f09780ad5c0 2024-03-29T11:22:48.071 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.094+0000 7f09767fc640 1 -- 172.21.15.47:0/3676270523 >> v1:172.21.15.47:6790/0 conn(0x7f097813a000 legacy=0x7f097814b880 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:22:48.071 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.094+0000 7f09767fc640 1 -- 172.21.15.47:0/3676270523 >> v1:172.21.15.121:6789/0 conn(0x7f09780acf60 legacy=0x7f097813ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:48.071 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.094+0000 7f09767fc640 10 monclient: _finish_hunting 0 2024-03-29T11:22:48.071 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.094+0000 7f09767fc640 1 monclient: found mon.noname-a 2024-03-29T11:22:48.071 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.094+0000 7f09767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:48.071 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.094+0000 7f09767fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-29T11:22:48.072 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.094+0000 7f09767fc640 1 -- 172.21.15.47:0/3676270523 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09780b4310 con 0x7f09780ad5c0 2024-03-29T11:22:48.072 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.094+0000 7f09767fc640 10 monclient: _finish_auth 0 2024-03-29T11:22:48.072 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.094+0000 7f09767fc640 10 monclient: _check_auth_tickets 2024-03-29T11:22:48.072 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.094+0000 7f09767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:48.072 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.094+0000 7f09767fc640 1 -- 172.21.15.47:0/3676270523 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f0968001690 con 0x7f09780ad5c0 2024-03-29T11:22:48.072 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.094+0000 7f09767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:48.072 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.094+0000 7f09767fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:48.072 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.094+0000 7f09767fc640 10 monclient: dump: 2024-03-29T11:22:48.072 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:22:48.072 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:22:48.072 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:48.072 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:48.072 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:48.072 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:22:48.072 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:22:48.072 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:22:48.073 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:22:48.073 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:22:48.073 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.094+0000 7f097daf4640 5 monclient: authenticate success, global_id 4566 2024-03-29T11:22:48.073 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.094+0000 7f097daf4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:48.075 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.096+0000 7f09767fc640 1 -- 172.21.15.47:0/3676270523 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09680017f0 con 0x7f09780ad5c0 2024-03-29T11:22:48.075 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.096+0000 7f09767fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:48.075 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.096+0000 7f09767fc640 1 -- 172.21.15.47:0/3676270523 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f0968001c00 con 0x7f09780ad5c0 2024-03-29T11:22:48.075 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.096+0000 7f09767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:48.075 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.096+0000 7f09767fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:48.075 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.096+0000 7f09767fc640 10 monclient: dump: 2024-03-29T11:22:48.075 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:22:48.075 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:22:48.075 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:48.075 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:48.076 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:48.076 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:22:48.076 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:22:48.076 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:22:48.076 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:22:48.076 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:22:48.076 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.096+0000 7f097daf4640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:48.076 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.096+0000 7f097daf4640 4 set_mon_vals no callback set 2024-03-29T11:22:48.076 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.097+0000 7f097daf4640 10 monclient: shutdown 2024-03-29T11:22:48.076 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.097+0000 7f097daf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:48.076 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.097+0000 7f097daf4640 1 -- 172.21.15.47:0/3676270523 >> v1:172.21.15.47:6789/0 conn(0x7f09780ad5c0 legacy=0x7f09781398f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:48.076 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.098+0000 7f097daf4640 1 -- 172.21.15.47:0/3676270523 shutdown_connections 2024-03-29T11:22:48.076 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.098+0000 7f097daf4640 1 -- 172.21.15.47:0/3676270523 >> 172.21.15.47:0/3676270523 conn(0x7f09780b35e0 msgr2=0x7f09780b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:48.077 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.098+0000 7f097daf4640 1 -- 172.21.15.47:0/3676270523 shutdown_connections 2024-03-29T11:22:48.077 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.098+0000 7f097daf4640 1 -- 172.21.15.47:0/3676270523 wait complete. 2024-03-29T11:22:48.078 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.099+0000 7f097daf4640 10 monclient: build_initial_monmap 2024-03-29T11:22:48.078 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.099+0000 7f097daf4640 10 monclient: monmap: 2024-03-29T11:22:48.078 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:22:48.079 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:48.079 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:22:48.079 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:22:48.079 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:48.079 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:22:48.079 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:22:48.079 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:22:48.079 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:22:48.079 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:22:48.079 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.100+0000 7f097daf4640 1 librados: starting msgr at 2024-03-29T11:22:48.079 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.100+0000 7f097daf4640 1 librados: starting objecter 2024-03-29T11:22:48.079 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.100+0000 7f097daf4640 1 Processor -- start 2024-03-29T11:22:48.079 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.100+0000 7f097daf4640 1 -- start start 2024-03-29T11:22:48.079 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.100+0000 7f097daf4640 1 librados: setting wanted keys 2024-03-29T11:22:48.079 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.100+0000 7f097daf4640 1 librados: calling monclient init 2024-03-29T11:22:48.079 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.100+0000 7f097daf4640 10 monclient: init 2024-03-29T11:22:48.080 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.101+0000 7f097daf4640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:48.080 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.101+0000 7f097daf4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:22:48.080 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.101+0000 7f097daf4640 10 monclient(hunting): picked mon.noname-b con 0x7f09780acf60 addr v1:172.21.15.121:6789/0 2024-03-29T11:22:48.080 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.101+0000 7f097daf4640 10 monclient(hunting): picked mon.noname-c con 0x7f09780ad5c0 addr v1:172.21.15.47:6790/0 2024-03-29T11:22:48.085 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.103+0000 7f097daf4640 10 monclient(hunting): picked mon.noname-a con 0x7f097813a000 addr v1:172.21.15.47:6789/0 2024-03-29T11:22:48.085 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.103+0000 7f097daf4640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f097813d2b0 con 0x7f097813a000 2024-03-29T11:22:48.085 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.103+0000 7f097daf4640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f097813d500 con 0x7f09780acf60 2024-03-29T11:22:48.085 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.103+0000 7f097daf4640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0978143830 con 0x7f09780ad5c0 2024-03-29T11:22:48.085 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.103+0000 7f097daf4640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:48.086 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.103+0000 7f097daf4640 10 monclient(hunting): authenticate will time out at 1411.291735s 2024-03-29T11:22:48.086 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.103+0000 7f0977fff640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f097813a000 0x7f0978070aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:38803/0 (socket says 172.21.15.47:38803) 2024-03-29T11:22:48.086 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.103+0000 7f0977fff640 1 -- 172.21.15.47:0/1590646184 learned_addr learned my addr 172.21.15.47:0/1590646184 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:22:48.086 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.104+0000 7f095ffff640 10 client.?.objecter ms_handle_connect 0x7f097813a000 2024-03-29T11:22:48.086 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.104+0000 7f095ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:48.086 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.104+0000 7f095ffff640 1 -- 172.21.15.47:0/1590646184 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3394462761 0 0) 0x7f097813d2b0 con 0x7f097813a000 2024-03-29T11:22:48.086 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.104+0000 7f095ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:48.086 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.104+0000 7f095ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:48.086 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.104+0000 7f095ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:48.086 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.104+0000 7f095ffff640 10 monclient(hunting): my global_id is 4569 2024-03-29T11:22:48.086 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.104+0000 7f095ffff640 1 -- 172.21.15.47:0/1590646184 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0964004eb0 con 0x7f097813a000 2024-03-29T11:22:48.086 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.104+0000 7f095ffff640 1 -- 172.21.15.47:0/1590646184 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1181718403 0 0) 0x7f0964004eb0 con 0x7f097813a000 2024-03-29T11:22:48.086 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.104+0000 7f095ffff640 1 -- 172.21.15.47:0/1590646184 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f097813d2b0 con 0x7f097813a000 2024-03-29T11:22:48.087 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.104+0000 7f095ffff640 1 -- 172.21.15.47:0/1590646184 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f0970001b90 con 0x7f097813a000 2024-03-29T11:22:48.087 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.104+0000 7f095ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:48.087 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.104+0000 7f095ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:48.087 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.104+0000 7f095ffff640 10 monclient(hunting): dump: 2024-03-29T11:22:48.087 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:22:48.087 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:22:48.087 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:48.087 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:48.087 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:48.087 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:22:48.087 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:22:48.087 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:22:48.087 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:22:48.087 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:22:48.087 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.104+0000 7f095ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:48.087 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.104+0000 7f095ffff640 1 -- 172.21.15.47:0/1590646184 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2967269262 0 0) 0x7f097813d2b0 con 0x7f097813a000 2024-03-29T11:22:48.088 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.104+0000 7f095ffff640 1 -- 172.21.15.47:0/1590646184 >> v1:172.21.15.47:6790/0 conn(0x7f09780ad5c0 legacy=0x7f097813bb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:22:48.088 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.104+0000 7f095ffff640 1 -- 172.21.15.47:0/1590646184 >> v1:172.21.15.121:6789/0 conn(0x7f09780acf60 legacy=0x7f097813b180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:22:48.088 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.105+0000 7f095ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:22:48.088 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.105+0000 7f095ffff640 1 monclient: found mon.a 2024-03-29T11:22:48.088 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.105+0000 7f095ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:48.088 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.105+0000 7f095ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:22:48.088 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.105+0000 7f095ffff640 1 -- 172.21.15.47:0/1590646184 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f097813d7e0 con 0x7f097813a000 2024-03-29T11:22:48.088 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.105+0000 7f095ffff640 10 monclient: _finish_auth 0 2024-03-29T11:22:48.088 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.105+0000 7f095ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:22:48.088 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.105+0000 7f095ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:48.088 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.105+0000 7f097daf4640 5 monclient: authenticate success, global_id 4569 2024-03-29T11:22:48.088 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.105+0000 7f097daf4640 10 monclient: _renew_subs 2024-03-29T11:22:48.088 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.105+0000 7f097daf4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:22:48.088 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.105+0000 7f097daf4640 1 -- 172.21.15.47:0/1590646184 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0978143d50 con 0x7f097813a000 2024-03-29T11:22:48.088 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.105+0000 7f097daf4640 10 client.4569.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:48.089 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.105+0000 7f097daf4640 10 monclient: _renew_subs 2024-03-29T11:22:48.089 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.105+0000 7f097daf4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:22:48.089 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.105+0000 7f097daf4640 1 -- 172.21.15.47:0/1590646184 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f097813c480 con 0x7f097813a000 2024-03-29T11:22:48.089 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.105+0000 7f097daf4640 1 librados: init done 2024-03-29T11:22:48.089 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.106+0000 7f095ffff640 1 -- 172.21.15.47:0/1590646184 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0970001d60 con 0x7f097813a000 2024-03-29T11:22:48.089 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.106+0000 7f095ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:48.089 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.106+0000 7f09757fa640 4 set_mon_vals no callback set 2024-03-29T11:22:48.089 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.106+0000 7f095ffff640 1 -- 172.21.15.47:0/1590646184 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f0970004dc0 con 0x7f097813a000 2024-03-29T11:22:48.089 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.106+0000 7f095ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:48.089 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.107+0000 7f095ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:48.089 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.107+0000 7f095ffff640 10 monclient: dump: 2024-03-29T11:22:48.089 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:22:48.089 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:22:48.090 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:48.090 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:48.090 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:48.090 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:22:48.090 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:22:48.090 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:22:48.090 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:22:48.090 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:22:48.090 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.107+0000 7f097daf4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:48.090 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.107+0000 7f097daf4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:48.090 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.107+0000 7f097daf4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:22:48.090 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.107+0000 7f097daf4640 1 -- 172.21.15.47:0/1590646184 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0978075f10 con 0x7f097813a000 2024-03-29T11:22:48.091 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.111+0000 7f095ffff640 1 -- 172.21.15.47:0/1590646184 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7f0970004f20 con 0x7f097813a000 2024-03-29T11:22:48.091 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.112+0000 7f095ffff640 1 -- 172.21.15.47:0/1590646184 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3418925575 0 0) 0x7f0970084df0 con 0x7f097813a000 2024-03-29T11:22:48.092 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.112+0000 7f095ffff640 10 client.4569.objecter ms_dispatch 0x7f0978000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:22:48.092 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.112+0000 7f095ffff640 3 client.4569.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:22:48.092 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.112+0000 7f095ffff640 3 client.4569.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:22:48.092 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.112+0000 7f095ffff640 20 client.4569.objecter dump_active .. 0 homeless 2024-03-29T11:22:48.092 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.112+0000 7f09777fe640 10 monclient: get_auth_request con 0x7f096406dd20 auth_method 0 2024-03-29T11:22:48.092 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.112+0000 7f095ffff640 10 client.4569.objecter ms_handle_connect 0x7f096406dd20 2024-03-29T11:22:48.093 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.115+0000 7f095ffff640 1 -- 172.21.15.47:0/1590646184 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0970050500 con 0x7f097813a000 2024-03-29T11:22:48.093 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.115+0000 7f095ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:48.093 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.115+0000 7f095ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:48.433 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.455+0000 7f097daf4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:48.433 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.455+0000 7f097daf4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:48.433 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.455+0000 7f097daf4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:22:48.433 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.455+0000 7f097daf4640 1 -- 172.21.15.47:0/1590646184 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f097806ab70 con 0x7f097813a000 2024-03-29T11:22:48.433 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.456+0000 7f095ffff640 1 -- 172.21.15.47:0/1590646184 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1178522771) 0x7f0970052600 con 0x7f097813a000 2024-03-29T11:22:48.433 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.456+0000 7f095ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:48.433 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.456+0000 7f095ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:48.434 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:22:48.434 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":11,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:22:45.925385+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:22:45.925385+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:48.438 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.460+0000 7f097daf4640 10 librados: watch_flush enter 2024-03-29T11:22:48.438 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.460+0000 7f097daf4640 10 librados: watch_flush exit 2024-03-29T11:22:48.438 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.460+0000 7f097daf4640 20 client.4569.objecter shutdown clearing up homeless session... 2024-03-29T11:22:48.438 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.460+0000 7f097daf4640 10 client.4569.objecter successfully canceled tick 2024-03-29T11:22:48.438 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.460+0000 7f097daf4640 1 -- 172.21.15.47:0/1590646184 >> v1:172.21.15.121:6816/1964117370 conn(0x7f096406dd20 legacy=0x7f09640701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:48.438 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.460+0000 7f097daf4640 10 monclient: shutdown 2024-03-29T11:22:48.438 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.460+0000 7f097daf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:48.438 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.460+0000 7f097daf4640 1 -- 172.21.15.47:0/1590646184 >> v1:172.21.15.47:6789/0 conn(0x7f097813a000 legacy=0x7f0978070aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:48.438 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.460+0000 7f097daf4640 1 -- 172.21.15.47:0/1590646184 shutdown_connections 2024-03-29T11:22:48.439 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.460+0000 7f097daf4640 1 -- 172.21.15.47:0/1590646184 >> 172.21.15.47:0/1590646184 conn(0x7f09780b35e0 msgr2=0x7f09780b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:48.439 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.461+0000 7f097daf4640 1 -- 172.21.15.47:0/1590646184 shutdown_connections 2024-03-29T11:22:48.439 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.461+0000 7f097daf4640 1 -- 172.21.15.47:0/1590646184 wait complete. 2024-03-29T11:22:48.439 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:48.461+0000 7f097daf4640 1 librados: shutdown 2024-03-29T11:22:48.455 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:55.262 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:55.421 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.444+0000 7f244592b640 10 monclient: get_monmap_and_config 2024-03-29T11:22:55.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.444+0000 7f244592b640 10 monclient: build_initial_monmap 2024-03-29T11:22:55.422 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.444+0000 7f244592b640 10 monclient: monmap: 2024-03-29T11:22:55.422 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:22:55.422 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:22:55.422 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:22:55.445011+0000 2024-03-29T11:22:55.422 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:22:55.445011+0000 2024-03-29T11:22:55.422 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:55.422 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:22:55.422 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:22:55.422 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:22:55.422 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:22:55.422 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:22:55.424 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.446+0000 7f244592b640 1 Processor -- start 2024-03-29T11:22:55.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.447+0000 7f244592b640 1 -- start start 2024-03-29T11:22:55.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.447+0000 7f244592b640 10 monclient: init 2024-03-29T11:22:55.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.448+0000 7f244592b640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:55.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.448+0000 7f244592b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:22:55.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.448+0000 7f244592b640 10 monclient(hunting): picked mon.noname-c con 0x7f24401422b0 addr v1:172.21.15.121:6789/0 2024-03-29T11:22:55.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.448+0000 7f244592b640 10 monclient(hunting): picked mon.noname-a con 0x7f2440142690 addr v1:172.21.15.47:6789/0 2024-03-29T11:22:55.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.448+0000 7f244592b640 10 monclient(hunting): picked mon.noname-b con 0x7f24400b17d0 addr v1:172.21.15.47:6790/0 2024-03-29T11:22:55.426 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.448+0000 7f244592b640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2440142c80 con 0x7f2440142690 2024-03-29T11:22:55.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.448+0000 7f244592b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24400aae40 con 0x7f24401422b0 2024-03-29T11:22:55.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.448+0000 7f244592b640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f244013fbd0 con 0x7f24400b17d0 2024-03-29T11:22:55.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.448+0000 7f244592b640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:55.427 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.448+0000 7f244592b640 10 monclient(hunting): authenticate will time out at 1418.636819s 2024-03-29T11:22:55.429 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.452+0000 7f2437fff640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f2440142690 0x7f24400b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:53629/0 (socket says 172.21.15.47:53629) 2024-03-29T11:22:55.430 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.452+0000 7f2437fff640 1 -- 172.21.15.47:0/2991741573 learned_addr learned my addr 172.21.15.47:0/2991741573 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:22:55.432 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.454+0000 7f243e7fc640 1 -- 172.21.15.47:0/2991741573 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3286714328 0 0) 0x7f244013fbd0 con 0x7f24400b17d0 2024-03-29T11:22:55.432 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.454+0000 7f243e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:55.432 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.454+0000 7f243e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:55.432 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.454+0000 7f243e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:55.432 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.454+0000 7f243e7fc640 10 monclient(hunting): my global_id is 4631 2024-03-29T11:22:55.432 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.454+0000 7f243e7fc640 1 -- 172.21.15.47:0/2991741573 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2420003520 con 0x7f24400b17d0 2024-03-29T11:22:55.432 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.454+0000 7f243e7fc640 1 -- 172.21.15.47:0/2991741573 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1056236470 0 0) 0x7f2440142c80 con 0x7f2440142690 2024-03-29T11:22:55.432 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.454+0000 7f243e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:55.432 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.454+0000 7f243e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:55.433 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.454+0000 7f243e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:55.433 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.454+0000 7f243e7fc640 10 monclient(hunting): my global_id is 4572 2024-03-29T11:22:55.433 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.454+0000 7f243e7fc640 1 -- 172.21.15.47:0/2991741573 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f244013fbd0 con 0x7f2440142690 2024-03-29T11:22:55.437 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.456+0000 7f243e7fc640 1 -- 172.21.15.47:0/2991741573 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 225864057 0 0) 0x7f2420003520 con 0x7f24400b17d0 2024-03-29T11:22:55.437 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.456+0000 7f243e7fc640 1 -- 172.21.15.47:0/2991741573 >> v1:172.21.15.121:6789/0 conn(0x7f24401422b0 legacy=0x7f244013f4c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:22:55.437 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.456+0000 7f243e7fc640 1 -- 172.21.15.47:0/2991741573 >> v1:172.21.15.47:6789/0 conn(0x7f2440142690 legacy=0x7f24400b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:55.437 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.456+0000 7f243e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:22:55.437 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.456+0000 7f243e7fc640 1 monclient: found mon.noname-b 2024-03-29T11:22:55.437 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.456+0000 7f243e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:55.437 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.456+0000 7f243e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-29T11:22:55.437 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.456+0000 7f243e7fc640 1 -- 172.21.15.47:0/2991741573 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2440151940 con 0x7f24400b17d0 2024-03-29T11:22:55.438 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.456+0000 7f243e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:22:55.438 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.457+0000 7f243e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:22:55.438 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.457+0000 7f243e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:55.438 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.457+0000 7f243e7fc640 1 -- 172.21.15.47:0/2991741573 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f2438001a10 con 0x7f24400b17d0 2024-03-29T11:22:55.438 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.457+0000 7f243e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:55.438 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.457+0000 7f243e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:55.438 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.457+0000 7f243e7fc640 10 monclient: dump: 2024-03-29T11:22:55.438 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:22:55.438 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:22:55.438 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:55.439 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:55.439 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:55.439 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:22:55.439 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:22:55.439 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:22:55.439 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:22:55.439 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:22:55.439 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.457+0000 7f244592b640 5 monclient: authenticate success, global_id 4631 2024-03-29T11:22:55.439 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.457+0000 7f244592b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:55.439 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.457+0000 7f243e7fc640 1 -- 172.21.15.47:0/2991741573 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2438001b70 con 0x7f24400b17d0 2024-03-29T11:22:55.439 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.457+0000 7f243e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:55.439 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.457+0000 7f244592b640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:55.440 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.457+0000 7f244592b640 4 set_mon_vals no callback set 2024-03-29T11:22:55.440 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.458+0000 7f243e7fc640 1 -- 172.21.15.47:0/2991741573 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f2438001f40 con 0x7f24400b17d0 2024-03-29T11:22:55.440 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.458+0000 7f243e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:55.440 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.458+0000 7f243e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:55.440 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.458+0000 7f243e7fc640 10 monclient: dump: 2024-03-29T11:22:55.440 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:22:55.440 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:22:55.440 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:55.440 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:55.440 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:55.440 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:22:55.441 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:22:55.441 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:22:55.441 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:22:55.441 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:22:55.441 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.458+0000 7f244592b640 10 monclient: shutdown 2024-03-29T11:22:55.441 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.458+0000 7f244592b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:55.441 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.458+0000 7f244592b640 1 -- 172.21.15.47:0/2991741573 >> v1:172.21.15.47:6790/0 conn(0x7f24400b17d0 legacy=0x7f24400b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:55.441 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.458+0000 7f244592b640 1 -- 172.21.15.47:0/2991741573 shutdown_connections 2024-03-29T11:22:55.441 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.458+0000 7f244592b640 1 -- 172.21.15.47:0/2991741573 >> 172.21.15.47:0/2991741573 conn(0x7f2440066a00 msgr2=0x7f2440066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:55.442 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.461+0000 7f244592b640 1 -- 172.21.15.47:0/2991741573 shutdown_connections 2024-03-29T11:22:55.443 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.463+0000 7f244592b640 1 -- 172.21.15.47:0/2991741573 wait complete. 2024-03-29T11:22:55.443 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.464+0000 7f244592b640 10 monclient: build_initial_monmap 2024-03-29T11:22:55.443 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.464+0000 7f244592b640 10 monclient: monmap: 2024-03-29T11:22:55.443 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:22:55.443 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:55.443 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:22:55.443 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:22:55.443 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:55.443 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:22:55.444 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:22:55.444 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:22:55.444 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:22:55.444 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:22:55.444 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.465+0000 7f244592b640 1 librados: starting msgr at 2024-03-29T11:22:55.444 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.465+0000 7f244592b640 1 librados: starting objecter 2024-03-29T11:22:55.444 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.465+0000 7f244592b640 1 Processor -- start 2024-03-29T11:22:55.444 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.465+0000 7f244592b640 1 -- start start 2024-03-29T11:22:55.444 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.465+0000 7f244592b640 1 librados: setting wanted keys 2024-03-29T11:22:55.444 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.465+0000 7f244592b640 1 librados: calling monclient init 2024-03-29T11:22:55.444 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.465+0000 7f244592b640 10 monclient: init 2024-03-29T11:22:55.446 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.466+0000 7f244592b640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:55.446 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.466+0000 7f244592b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:22:55.446 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.466+0000 7f244592b640 10 monclient(hunting): picked mon.noname-c con 0x7f24400b17d0 addr v1:172.21.15.47:6790/0 2024-03-29T11:22:55.446 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.466+0000 7f244592b640 10 monclient(hunting): picked mon.noname-b con 0x7f24401422b0 addr v1:172.21.15.121:6789/0 2024-03-29T11:22:55.446 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.466+0000 7f244592b640 10 monclient(hunting): picked mon.noname-a con 0x7f2440142690 addr v1:172.21.15.47:6789/0 2024-03-29T11:22:55.446 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.466+0000 7f244592b640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f244006de90 con 0x7f2440142690 2024-03-29T11:22:55.446 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.466+0000 7f244592b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24400b6f00 con 0x7f24401422b0 2024-03-29T11:22:55.446 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.466+0000 7f244592b640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2440071ec0 con 0x7f24400b17d0 2024-03-29T11:22:55.446 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.466+0000 7f244592b640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:55.446 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.466+0000 7f244592b640 10 monclient(hunting): authenticate will time out at 1418.654250s 2024-03-29T11:22:55.447 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.469+0000 7f243f7fe640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f2440142690 0x7f24401ddf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:54291/0 (socket says 172.21.15.47:54291) 2024-03-29T11:22:55.447 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.469+0000 7f243f7fe640 1 -- 172.21.15.47:0/3987577966 learned_addr learned my addr 172.21.15.47:0/3987577966 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:22:55.447 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.469+0000 7f243cff9640 10 client.?.objecter ms_handle_connect 0x7f2440142690 2024-03-29T11:22:55.448 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.469+0000 7f243cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:55.451 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.470+0000 7f243cff9640 10 client.?.objecter ms_handle_connect 0x7f24400b17d0 2024-03-29T11:22:55.451 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.470+0000 7f243cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:55.451 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.470+0000 7f243cff9640 1 -- 172.21.15.47:0/3987577966 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 360256677 0 0) 0x7f2440071ec0 con 0x7f24400b17d0 2024-03-29T11:22:55.451 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.470+0000 7f243cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:22:55.451 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.470+0000 7f243cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:55.451 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.470+0000 7f243cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:55.452 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.470+0000 7f243cff9640 10 monclient(hunting): my global_id is 4634 2024-03-29T11:22:55.452 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.470+0000 7f243cff9640 1 -- 172.21.15.47:0/3987577966 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2424004eb0 con 0x7f24400b17d0 2024-03-29T11:22:55.452 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.471+0000 7f243cff9640 1 -- 172.21.15.47:0/3987577966 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3191939187 0 0) 0x7f2424004eb0 con 0x7f24400b17d0 2024-03-29T11:22:55.452 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.471+0000 7f243cff9640 1 -- 172.21.15.47:0/3987577966 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2424005300 con 0x7f24400b17d0 2024-03-29T11:22:55.452 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.471+0000 7f243cff9640 1 -- 172.21.15.47:0/3987577966 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f2428001830 con 0x7f24400b17d0 2024-03-29T11:22:55.452 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.471+0000 7f243cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:55.452 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.471+0000 7f243cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:55.452 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.471+0000 7f243cff9640 10 monclient(hunting): dump: 2024-03-29T11:22:55.452 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:22:55.452 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:22:55.452 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:55.452 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:55.453 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:55.453 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:22:55.453 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:22:55.453 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:22:55.453 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:22:55.453 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:22:55.453 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.471+0000 7f243cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:55.455 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.475+0000 7f243cff9640 10 client.?.objecter ms_handle_connect 0x7f24401422b0 2024-03-29T11:22:55.455 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.475+0000 7f243cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:55.455 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.475+0000 7f243cff9640 1 -- 172.21.15.47:0/3987577966 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 455273373 0 0) 0x7f244006de90 con 0x7f2440142690 2024-03-29T11:22:55.456 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.475+0000 7f243cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:22:55.456 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.475+0000 7f243cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:55.456 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.475+0000 7f243cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:55.456 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.475+0000 7f243cff9640 10 monclient(hunting): my global_id is 4575 2024-03-29T11:22:55.456 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.475+0000 7f243cff9640 1 -- 172.21.15.47:0/3987577966 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2424004eb0 con 0x7f2440142690 2024-03-29T11:22:55.456 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.475+0000 7f243cff9640 1 -- 172.21.15.47:0/3987577966 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2208335347 0 0) 0x7f2424005300 con 0x7f24400b17d0 2024-03-29T11:22:55.456 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.475+0000 7f243cff9640 1 -- 172.21.15.47:0/3987577966 >> v1:172.21.15.121:6789/0 conn(0x7f24401422b0 legacy=0x7f24401da800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:55.456 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.475+0000 7f243cff9640 1 -- 172.21.15.47:0/3987577966 >> v1:172.21.15.47:6789/0 conn(0x7f2440142690 legacy=0x7f24401ddf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:55.456 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.475+0000 7f243cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:22:55.456 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.476+0000 7f243cff9640 1 monclient: found mon.c 2024-03-29T11:22:55.456 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.476+0000 7f243cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:55.456 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.476+0000 7f243cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:22:55.457 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.476+0000 7f243cff9640 1 -- 172.21.15.47:0/3987577966 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f244012c600 con 0x7f24400b17d0 2024-03-29T11:22:55.457 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.476+0000 7f243cff9640 10 monclient: _finish_auth 0 2024-03-29T11:22:55.457 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.476+0000 7f243cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:22:55.457 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.476+0000 7f243cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:55.464 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.477+0000 7f244592b640 5 monclient: authenticate success, global_id 4634 2024-03-29T11:22:55.464 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.477+0000 7f244592b640 10 monclient: _renew_subs 2024-03-29T11:22:55.464 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.477+0000 7f244592b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:22:55.464 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.477+0000 7f244592b640 1 -- 172.21.15.47:0/3987577966 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24400b7400 con 0x7f24400b17d0 2024-03-29T11:22:55.464 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.477+0000 7f244592b640 10 client.4634.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:55.464 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.477+0000 7f244592b640 10 monclient: _renew_subs 2024-03-29T11:22:55.464 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.477+0000 7f244592b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:22:55.464 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.477+0000 7f244592b640 1 -- 172.21.15.47:0/3987577966 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f24400b6bc0 con 0x7f24400b17d0 2024-03-29T11:22:55.464 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.477+0000 7f243cff9640 1 -- 172.21.15.47:0/3987577966 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2428001dc0 con 0x7f24400b17d0 2024-03-29T11:22:55.464 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.477+0000 7f243cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:55.464 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.478+0000 7f244592b640 1 librados: init done 2024-03-29T11:22:55.465 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.478+0000 7f243cff9640 1 -- 172.21.15.47:0/3987577966 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f2428003830 con 0x7f24400b17d0 2024-03-29T11:22:55.465 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.478+0000 7f243cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:55.465 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.478+0000 7f243cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:55.465 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.478+0000 7f243cff9640 10 monclient: dump: 2024-03-29T11:22:55.465 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:22:55.465 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:22:55.465 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:55.465 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:22:55.465 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:55.465 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:22:55.465 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:22:55.466 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:22:55.466 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:22:55.466 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:22:55.466 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.478+0000 7f243dffb640 4 set_mon_vals no callback set 2024-03-29T11:22:55.466 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.479+0000 7f244592b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:55.466 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.479+0000 7f244592b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:55.466 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.479+0000 7f244592b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:22:55.466 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.479+0000 7f244592b640 1 -- 172.21.15.47:0/3987577966 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2418005080 con 0x7f24400b17d0 2024-03-29T11:22:55.466 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.481+0000 7f243cff9640 1 -- 172.21.15.47:0/3987577966 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7f2428019800 con 0x7f24400b17d0 2024-03-29T11:22:55.466 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.482+0000 7f243cff9640 1 -- 172.21.15.47:0/3987577966 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3418925575 0 0) 0x7f2428083750 con 0x7f24400b17d0 2024-03-29T11:22:55.466 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.482+0000 7f243cff9640 10 client.4634.objecter ms_dispatch 0x7f2440000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:22:55.466 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.482+0000 7f243cff9640 3 client.4634.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:22:55.467 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.482+0000 7f243cff9640 3 client.4634.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:22:55.467 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.482+0000 7f243cff9640 20 client.4634.objecter dump_active .. 0 homeless 2024-03-29T11:22:55.467 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.483+0000 7f2437fff640 10 monclient: get_auth_request con 0x7f242406df30 auth_method 0 2024-03-29T11:22:55.467 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.484+0000 7f243cff9640 10 client.4634.objecter ms_handle_connect 0x7f242406df30 2024-03-29T11:22:55.467 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.489+0000 7f243cff9640 1 -- 172.21.15.47:0/3987577966 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f242804ee60 con 0x7f24400b17d0 2024-03-29T11:22:55.468 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.489+0000 7f243cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:55.468 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.489+0000 7f243cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:55.788 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.810+0000 7f244592b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:55.788 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.810+0000 7f244592b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:55.788 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.810+0000 7f244592b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:22:55.788 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.810+0000 7f244592b640 1 -- 172.21.15.47:0/3987577966 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2418005330 con 0x7f24400b17d0 2024-03-29T11:22:55.789 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:22:55.789 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":11,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:22:45.925385+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:22:45.925385+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:55.789 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.811+0000 7f243cff9640 1 -- 172.21.15.47:0/3987577966 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1178522771) 0x7f2428050f60 con 0x7f24400b17d0 2024-03-29T11:22:55.789 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.811+0000 7f243cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:55.790 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.811+0000 7f243cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:55.792 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.815+0000 7f244592b640 10 librados: watch_flush enter 2024-03-29T11:22:55.794 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.816+0000 7f244592b640 10 librados: watch_flush exit 2024-03-29T11:22:55.794 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.816+0000 7f244592b640 20 client.4634.objecter shutdown clearing up homeless session... 2024-03-29T11:22:55.794 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.816+0000 7f244592b640 10 client.4634.objecter successfully canceled tick 2024-03-29T11:22:55.794 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.816+0000 7f244592b640 1 -- 172.21.15.47:0/3987577966 >> v1:172.21.15.121:6816/1964117370 conn(0x7f242406df30 legacy=0x7f24240703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:55.794 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.816+0000 7f244592b640 10 monclient: shutdown 2024-03-29T11:22:55.795 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.816+0000 7f244592b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:55.795 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.816+0000 7f244592b640 1 -- 172.21.15.47:0/3987577966 >> v1:172.21.15.47:6790/0 conn(0x7f24400b17d0 legacy=0x7f244012bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:55.795 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.816+0000 7f244592b640 1 -- 172.21.15.47:0/3987577966 shutdown_connections 2024-03-29T11:22:55.795 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.816+0000 7f244592b640 1 -- 172.21.15.47:0/3987577966 >> 172.21.15.47:0/3987577966 conn(0x7f2440066a00 msgr2=0x7f244013c1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:55.795 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.816+0000 7f244592b640 1 -- 172.21.15.47:0/3987577966 shutdown_connections 2024-03-29T11:22:55.795 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.816+0000 7f244592b640 1 -- 172.21.15.47:0/3987577966 wait complete. 2024-03-29T11:22:55.795 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:22:55.817+0000 7f244592b640 1 librados: shutdown 2024-03-29T11:22:55.813 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:02.619 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:02.771 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.793+0000 7ffbf8ac9640 10 monclient: get_monmap_and_config 2024-03-29T11:23:02.771 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.793+0000 7ffbf8ac9640 10 monclient: build_initial_monmap 2024-03-29T11:23:02.771 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.793+0000 7ffbf8ac9640 10 monclient: monmap: 2024-03-29T11:23:02.772 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:23:02.772 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:23:02.772 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:23:02.794885+0000 2024-03-29T11:23:02.772 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:23:02.794885+0000 2024-03-29T11:23:02.772 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:02.772 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:23:02.772 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:23:02.772 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:23:02.772 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:23:02.772 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:23:02.773 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.794+0000 7ffbf8ac9640 1 Processor -- start 2024-03-29T11:23:02.773 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.794+0000 7ffbf8ac9640 1 -- start start 2024-03-29T11:23:02.773 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.794+0000 7ffbf8ac9640 10 monclient: init 2024-03-29T11:23:02.773 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.796+0000 7ffbf8ac9640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:02.773 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.796+0000 7ffbf8ac9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:23:02.774 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.796+0000 7ffbf8ac9640 10 monclient(hunting): picked mon.noname-c con 0x7ffbf41457f0 addr v1:172.21.15.121:6789/0 2024-03-29T11:23:02.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.796+0000 7ffbf8ac9640 10 monclient(hunting): picked mon.noname-a con 0x7ffbf413f650 addr v1:172.21.15.47:6789/0 2024-03-29T11:23:02.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.797+0000 7ffbf8ac9640 10 monclient(hunting): picked mon.noname-b con 0x7ffbf40a9d70 addr v1:172.21.15.47:6790/0 2024-03-29T11:23:02.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.797+0000 7ffbf8ac9640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbf40aaef0 con 0x7ffbf413f650 2024-03-29T11:23:02.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.797+0000 7ffbf8ac9640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbf4145bd0 con 0x7ffbf41457f0 2024-03-29T11:23:02.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.797+0000 7ffbf8ac9640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbf4145eb0 con 0x7ffbf40a9d70 2024-03-29T11:23:02.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.797+0000 7ffbf8ac9640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:02.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.797+0000 7ffbf8ac9640 10 monclient(hunting): authenticate will time out at 1425.985088s 2024-03-29T11:23:02.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.797+0000 7ffbf2d9d640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7ffbf40a9d70 0x7ffbf414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:60594/0 (socket says 172.21.15.47:60594) 2024-03-29T11:23:02.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.797+0000 7ffbf2d9d640 1 -- 172.21.15.47:0/2491110315 learned_addr learned my addr 172.21.15.47:0/2491110315 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:23:02.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.797+0000 7ffbf159a640 1 -- 172.21.15.47:0/2491110315 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 973072977 0 0) 0x7ffbf4145eb0 con 0x7ffbf40a9d70 2024-03-29T11:23:02.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.797+0000 7ffbf159a640 10 monclient(hunting): _negotiate 2024-03-29T11:23:02.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.797+0000 7ffbf159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:02.776 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.797+0000 7ffbf159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:02.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.797+0000 7ffbf159a640 10 monclient(hunting): my global_id is 4637 2024-03-29T11:23:02.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.797+0000 7ffbf159a640 1 -- 172.21.15.47:0/2491110315 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbdc003520 con 0x7ffbf40a9d70 2024-03-29T11:23:02.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.798+0000 7ffbf159a640 1 -- 172.21.15.47:0/2491110315 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2774263153 0 0) 0x7ffbdc003520 con 0x7ffbf40a9d70 2024-03-29T11:23:02.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.798+0000 7ffbf159a640 1 -- 172.21.15.47:0/2491110315 >> v1:172.21.15.121:6789/0 conn(0x7ffbf41457f0 legacy=0x7ffbf413ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:23:02.777 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.799+0000 7ffbf159a640 1 -- 172.21.15.47:0/2491110315 >> v1:172.21.15.47:6789/0 conn(0x7ffbf413f650 legacy=0x7ffbf40b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:23:02.779 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.800+0000 7ffbf159a640 10 monclient: _finish_hunting 0 2024-03-29T11:23:02.779 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.800+0000 7ffbf159a640 1 monclient: found mon.noname-b 2024-03-29T11:23:02.780 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.800+0000 7ffbf159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:02.780 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.800+0000 7ffbf159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-29T11:23:02.780 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.800+0000 7ffbf159a640 1 -- 172.21.15.47:0/2491110315 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbf4146340 con 0x7ffbf40a9d70 2024-03-29T11:23:02.780 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.800+0000 7ffbf159a640 10 monclient: _finish_auth 0 2024-03-29T11:23:02.780 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.800+0000 7ffbf159a640 10 monclient: _check_auth_tickets 2024-03-29T11:23:02.780 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.800+0000 7ffbf159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:02.780 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.800+0000 7ffbf159a640 1 -- 172.21.15.47:0/2491110315 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7ffbec001a10 con 0x7ffbf40a9d70 2024-03-29T11:23:02.780 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.800+0000 7ffbf8ac9640 5 monclient: authenticate success, global_id 4637 2024-03-29T11:23:02.780 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.800+0000 7ffbf8ac9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:02.780 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.800+0000 7ffbf159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:02.780 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.800+0000 7ffbf159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:02.780 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.800+0000 7ffbf159a640 10 monclient: dump: 2024-03-29T11:23:02.780 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:23:02.780 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:23:02.781 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:23:02.781 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:23:02.781 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:02.781 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:23:02.781 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:23:02.781 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:23:02.781 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:23:02.781 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:23:02.781 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.800+0000 7ffbf8ac9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:02.781 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.801+0000 7ffbf159a640 1 -- 172.21.15.47:0/2491110315 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffbec001be0 con 0x7ffbf40a9d70 2024-03-29T11:23:02.781 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.801+0000 7ffbf159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:02.781 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.801+0000 7ffbf159a640 1 -- 172.21.15.47:0/2491110315 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7ffbec002030 con 0x7ffbf40a9d70 2024-03-29T11:23:02.781 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.801+0000 7ffbf159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:02.781 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.801+0000 7ffbf159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:02.781 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.801+0000 7ffbf159a640 10 monclient: dump: 2024-03-29T11:23:02.782 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:23:02.782 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:23:02.782 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:23:02.782 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:23:02.782 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:02.782 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:23:02.782 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:23:02.782 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:23:02.782 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:23:02.782 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:23:02.782 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.801+0000 7ffbf8ac9640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:02.782 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.801+0000 7ffbf8ac9640 4 set_mon_vals no callback set 2024-03-29T11:23:02.782 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.802+0000 7ffbf8ac9640 10 monclient: shutdown 2024-03-29T11:23:02.782 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.802+0000 7ffbf8ac9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:02.783 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.802+0000 7ffbf8ac9640 1 -- 172.21.15.47:0/2491110315 >> v1:172.21.15.47:6790/0 conn(0x7ffbf40a9d70 legacy=0x7ffbf414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:02.783 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.802+0000 7ffbf8ac9640 1 -- 172.21.15.47:0/2491110315 shutdown_connections 2024-03-29T11:23:02.783 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.802+0000 7ffbf8ac9640 1 -- 172.21.15.47:0/2491110315 >> 172.21.15.47:0/2491110315 conn(0x7ffbf4066a00 msgr2=0x7ffbf4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:02.783 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.804+0000 7ffbf8ac9640 1 -- 172.21.15.47:0/2491110315 shutdown_connections 2024-03-29T11:23:02.783 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.805+0000 7ffbf8ac9640 1 -- 172.21.15.47:0/2491110315 wait complete. 2024-03-29T11:23:02.784 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.806+0000 7ffbf8ac9640 10 monclient: build_initial_monmap 2024-03-29T11:23:02.784 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.806+0000 7ffbf8ac9640 10 monclient: monmap: 2024-03-29T11:23:02.784 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:23:02.784 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:02.784 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:23:02.784 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:23:02.784 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:02.784 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:23:02.785 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:23:02.785 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:23:02.785 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:23:02.785 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:23:02.786 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.807+0000 7ffbf8ac9640 1 librados: starting msgr at 2024-03-29T11:23:02.786 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.807+0000 7ffbf8ac9640 1 librados: starting objecter 2024-03-29T11:23:02.786 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.807+0000 7ffbf8ac9640 1 Processor -- start 2024-03-29T11:23:02.786 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.807+0000 7ffbf8ac9640 1 -- start start 2024-03-29T11:23:02.786 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.807+0000 7ffbf8ac9640 1 librados: setting wanted keys 2024-03-29T11:23:02.786 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.807+0000 7ffbf8ac9640 1 librados: calling monclient init 2024-03-29T11:23:02.786 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.808+0000 7ffbf8ac9640 10 monclient: init 2024-03-29T11:23:02.787 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.808+0000 7ffbf8ac9640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:02.787 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.808+0000 7ffbf8ac9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:23:02.787 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.809+0000 7ffbf8ac9640 10 monclient(hunting): picked mon.noname-b con 0x7ffbf40a9d70 addr v1:172.21.15.121:6789/0 2024-03-29T11:23:02.788 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.809+0000 7ffbf8ac9640 10 monclient(hunting): picked mon.noname-c con 0x7ffbf413f650 addr v1:172.21.15.47:6790/0 2024-03-29T11:23:02.788 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.809+0000 7ffbf8ac9640 10 monclient(hunting): picked mon.noname-a con 0x7ffbf41457f0 addr v1:172.21.15.47:6789/0 2024-03-29T11:23:02.788 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.809+0000 7ffbf8ac9640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbf413e3a0 con 0x7ffbf41457f0 2024-03-29T11:23:02.788 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.809+0000 7ffbf8ac9640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbf413e120 con 0x7ffbf40a9d70 2024-03-29T11:23:02.788 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.809+0000 7ffbf8ac9640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbf413cd20 con 0x7ffbf413f650 2024-03-29T11:23:02.788 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.809+0000 7ffbf8ac9640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:02.788 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.809+0000 7ffbf8ac9640 10 monclient(hunting): authenticate will time out at 1425.997910s 2024-03-29T11:23:02.793 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.814+0000 7ffbf259c640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7ffbf40a9d70 0x7ffbf413c610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.47:36185/0 (socket says 172.21.15.47:36185) 2024-03-29T11:23:02.793 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.814+0000 7ffbf259c640 1 -- 172.21.15.47:0/1932109204 learned_addr learned my addr 172.21.15.47:0/1932109204 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:23:02.793 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.814+0000 7ffbe2ffd640 10 client.?.objecter ms_handle_connect 0x7ffbf41457f0 2024-03-29T11:23:02.793 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.814+0000 7ffbe2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:02.793 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.815+0000 7ffbe2ffd640 1 -- 172.21.15.47:0/1932109204 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3486786736 0 0) 0x7ffbf413e3a0 con 0x7ffbf41457f0 2024-03-29T11:23:02.793 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.815+0000 7ffbe2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:02.793 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.815+0000 7ffbe2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:02.793 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.815+0000 7ffbe2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:02.793 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.815+0000 7ffbe2ffd640 10 monclient(hunting): my global_id is 4578 2024-03-29T11:23:02.793 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.815+0000 7ffbe2ffd640 1 -- 172.21.15.47:0/1932109204 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbd4004eb0 con 0x7ffbf41457f0 2024-03-29T11:23:02.794 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.817+0000 7ffbe2ffd640 10 client.?.objecter ms_handle_connect 0x7ffbf40a9d70 2024-03-29T11:23:02.794 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.817+0000 7ffbe2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:02.795 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.817+0000 7ffbe2ffd640 1 -- 172.21.15.47:0/1932109204 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2284687691 0 0) 0x7ffbf413e120 con 0x7ffbf40a9d70 2024-03-29T11:23:02.795 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.817+0000 7ffbe2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:02.795 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.817+0000 7ffbe2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:02.795 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.817+0000 7ffbe2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:02.795 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.817+0000 7ffbe2ffd640 10 monclient(hunting): my global_id is 4549 2024-03-29T11:23:02.795 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.817+0000 7ffbe2ffd640 1 -- 172.21.15.47:0/1932109204 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbf413e3a0 con 0x7ffbf40a9d70 2024-03-29T11:23:02.797 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.818+0000 7ffbe2ffd640 1 -- 172.21.15.47:0/1932109204 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1279187756 0 0) 0x7ffbd4004eb0 con 0x7ffbf41457f0 2024-03-29T11:23:02.798 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.819+0000 7ffbe2ffd640 1 -- 172.21.15.47:0/1932109204 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffbf413e120 con 0x7ffbf41457f0 2024-03-29T11:23:02.798 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.819+0000 7ffbe2ffd640 1 -- 172.21.15.47:0/1932109204 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7ffbec001ef0 con 0x7ffbf41457f0 2024-03-29T11:23:02.798 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.819+0000 7ffbe2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:02.798 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.819+0000 7ffbe2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:23:02.798 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.819+0000 7ffbe2ffd640 10 monclient(hunting): dump: 2024-03-29T11:23:02.798 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:23:02.798 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:23:02.798 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:23:02.798 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:23:02.798 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:02.798 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:23:02.798 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:23:02.798 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:23:02.799 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:23:02.799 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:23:02.799 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.819+0000 7ffbe2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:02.799 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.819+0000 7ffbe2ffd640 1 -- 172.21.15.47:0/1932109204 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3226516612 0 0) 0x7ffbf413e3a0 con 0x7ffbf40a9d70 2024-03-29T11:23:02.799 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.819+0000 7ffbe2ffd640 1 -- 172.21.15.47:0/1932109204 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffbd4004eb0 con 0x7ffbf40a9d70 2024-03-29T11:23:02.799 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.819+0000 7ffbe2ffd640 1 -- 172.21.15.47:0/1932109204 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7ffbe4001790 con 0x7ffbf40a9d70 2024-03-29T11:23:02.799 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.819+0000 7ffbe2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:02.799 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.819+0000 7ffbe2ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:02.799 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.819+0000 7ffbe2ffd640 10 monclient(hunting): dump: 2024-03-29T11:23:02.799 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:23:02.799 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:23:02.799 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:23:02.799 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:23:02.800 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:02.800 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:23:02.800 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:23:02.800 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:23:02.800 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:23:02.800 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:23:02.800 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.819+0000 7ffbe2ffd640 10 client.?.objecter ms_handle_connect 0x7ffbf413f650 2024-03-29T11:23:02.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.819+0000 7ffbe2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:02.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.822+0000 7ffbe2ffd640 1 -- 172.21.15.47:0/1932109204 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1046853039 0 0) 0x7ffbf413cd20 con 0x7ffbf413f650 2024-03-29T11:23:02.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.822+0000 7ffbe2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:02.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.822+0000 7ffbe2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:02.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.822+0000 7ffbe2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:02.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.822+0000 7ffbe2ffd640 10 monclient(hunting): my global_id is 4640 2024-03-29T11:23:02.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.822+0000 7ffbe2ffd640 1 -- 172.21.15.47:0/1932109204 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbf413e3a0 con 0x7ffbf413f650 2024-03-29T11:23:02.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.822+0000 7ffbe2ffd640 1 -- 172.21.15.47:0/1932109204 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4080822431 0 0) 0x7ffbf413e120 con 0x7ffbf41457f0 2024-03-29T11:23:02.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.822+0000 7ffbe2ffd640 1 -- 172.21.15.47:0/1932109204 >> v1:172.21.15.47:6790/0 conn(0x7ffbf413f650 legacy=0x7ffbf41dedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:02.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.822+0000 7ffbe2ffd640 1 -- 172.21.15.47:0/1932109204 >> v1:172.21.15.121:6789/0 conn(0x7ffbf40a9d70 legacy=0x7ffbf413c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:02.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.822+0000 7ffbe2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:23:02.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.822+0000 7ffbe2ffd640 1 monclient: found mon.a 2024-03-29T11:23:02.801 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.822+0000 7ffbe2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:02.802 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.822+0000 7ffbe2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:23:02.802 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.822+0000 7ffbe2ffd640 1 -- 172.21.15.47:0/1932109204 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbf4141450 con 0x7ffbf41457f0 2024-03-29T11:23:02.802 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.822+0000 7ffbe2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:23:02.802 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.822+0000 7ffbe2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:23:02.802 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.822+0000 7ffbe2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:02.802 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.822+0000 7ffbe2ffd640 1 -- 172.21.15.47:0/1932109204 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffbec001390 con 0x7ffbf41457f0 2024-03-29T11:23:02.802 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.822+0000 7ffbe2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:02.802 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.822+0000 7ffbe2ffd640 1 -- 172.21.15.47:0/1932109204 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7ffbec004f70 con 0x7ffbf41457f0 2024-03-29T11:23:02.802 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.822+0000 7ffbe2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:02.802 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.822+0000 7ffbe2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:02.802 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.822+0000 7ffbe2ffd640 10 monclient: dump: 2024-03-29T11:23:02.802 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:23:02.803 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:23:02.803 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:23:02.803 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:23:02.803 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:02.803 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:23:02.803 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:23:02.803 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:23:02.803 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:23:02.803 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:23:02.804 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.822+0000 7ffbf0d99640 4 set_mon_vals no callback set 2024-03-29T11:23:02.804 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.823+0000 7ffbf8ac9640 5 monclient: authenticate success, global_id 4578 2024-03-29T11:23:02.804 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.823+0000 7ffbf8ac9640 10 monclient: _renew_subs 2024-03-29T11:23:02.804 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.823+0000 7ffbf8ac9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:23:02.804 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.823+0000 7ffbf8ac9640 1 -- 172.21.15.47:0/1932109204 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbf4141730 con 0x7ffbf41457f0 2024-03-29T11:23:02.804 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.823+0000 7ffbf8ac9640 10 client.4578.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:02.804 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.823+0000 7ffbf8ac9640 10 monclient: _renew_subs 2024-03-29T11:23:02.804 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.823+0000 7ffbf8ac9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:23:02.804 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.823+0000 7ffbf8ac9640 1 -- 172.21.15.47:0/1932109204 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbf4142b60 con 0x7ffbf41457f0 2024-03-29T11:23:02.805 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.823+0000 7ffbf8ac9640 1 librados: init done 2024-03-29T11:23:02.805 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.825+0000 7ffbf8ac9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:02.805 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.825+0000 7ffbf8ac9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:02.805 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.825+0000 7ffbf8ac9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:23:02.805 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.825+0000 7ffbf8ac9640 1 -- 172.21.15.47:0/1932109204 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbf4142eb0 con 0x7ffbf41457f0 2024-03-29T11:23:02.805 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.825+0000 7ffbe2ffd640 1 -- 172.21.15.47:0/1932109204 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7ffbec01af40 con 0x7ffbf41457f0 2024-03-29T11:23:02.805 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.826+0000 7ffbe2ffd640 1 -- 172.21.15.47:0/1932109204 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3418925575 0 0) 0x7ffbec084db0 con 0x7ffbf41457f0 2024-03-29T11:23:02.807 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.826+0000 7ffbe2ffd640 10 client.4578.objecter ms_dispatch 0x7ffbf4000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:23:02.807 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.826+0000 7ffbe2ffd640 3 client.4578.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:23:02.807 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.826+0000 7ffbe2ffd640 3 client.4578.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:23:02.807 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.826+0000 7ffbe2ffd640 20 client.4578.objecter dump_active .. 0 homeless 2024-03-29T11:23:02.808 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.829+0000 7ffbf259c640 10 monclient: get_auth_request con 0x7ffbd4076870 auth_method 0 2024-03-29T11:23:02.808 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.829+0000 7ffbe2ffd640 10 client.4578.objecter ms_handle_connect 0x7ffbd4076870 2024-03-29T11:23:02.812 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.834+0000 7ffbe2ffd640 1 -- 172.21.15.47:0/1932109204 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffbec050690 con 0x7ffbf41457f0 2024-03-29T11:23:02.812 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.834+0000 7ffbe2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:02.812 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:02.834+0000 7ffbe2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:03.149 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:03.171+0000 7ffbf8ac9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:03.150 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:03.171+0000 7ffbf8ac9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:03.150 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:03.171+0000 7ffbf8ac9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:23:03.150 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:03.171+0000 7ffbf8ac9640 1 -- 172.21.15.47:0/1932109204 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffbf4075f10 con 0x7ffbf41457f0 2024-03-29T11:23:03.151 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:03.173+0000 7ffbe2ffd640 1 -- 172.21.15.47:0/1932109204 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1178522771) 0x7ffbec0525c0 con 0x7ffbf41457f0 2024-03-29T11:23:03.151 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:03.173+0000 7ffbe2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:03.151 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:03.173+0000 7ffbe2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:03.152 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:23:03.152 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":11,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:22:45.925385+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:22:45.925385+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:03.155 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:03.177+0000 7ffbf8ac9640 10 librados: watch_flush enter 2024-03-29T11:23:03.155 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:03.177+0000 7ffbf8ac9640 10 librados: watch_flush exit 2024-03-29T11:23:03.155 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:03.177+0000 7ffbf8ac9640 20 client.4578.objecter shutdown clearing up homeless session... 2024-03-29T11:23:03.155 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:03.177+0000 7ffbf8ac9640 10 client.4578.objecter successfully canceled tick 2024-03-29T11:23:03.156 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:03.178+0000 7ffbf8ac9640 1 -- 172.21.15.47:0/1932109204 >> v1:172.21.15.121:6816/1964117370 conn(0x7ffbd4076870 legacy=0x7ffbd4078d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:03.156 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:03.178+0000 7ffbf8ac9640 10 monclient: shutdown 2024-03-29T11:23:03.156 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:03.178+0000 7ffbf8ac9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:03.156 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:03.178+0000 7ffbf8ac9640 1 -- 172.21.15.47:0/1932109204 >> v1:172.21.15.47:6789/0 conn(0x7ffbf41457f0 legacy=0x7ffbf41e2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:03.156 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:03.178+0000 7ffbf8ac9640 1 -- 172.21.15.47:0/1932109204 shutdown_connections 2024-03-29T11:23:03.156 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:03.178+0000 7ffbf8ac9640 1 -- 172.21.15.47:0/1932109204 >> 172.21.15.47:0/1932109204 conn(0x7ffbf4066a00 msgr2=0x7ffbf40b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:03.156 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:03.178+0000 7ffbf8ac9640 1 -- 172.21.15.47:0/1932109204 shutdown_connections 2024-03-29T11:23:03.156 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:03.178+0000 7ffbf8ac9640 1 -- 172.21.15.47:0/1932109204 wait complete. 2024-03-29T11:23:03.156 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:03.178+0000 7ffbf8ac9640 1 librados: shutdown 2024-03-29T11:23:03.172 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:09.978 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:10.154 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.174+0000 7fbd3dc01640 10 monclient: get_monmap_and_config 2024-03-29T11:23:10.154 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.174+0000 7fbd3dc01640 10 monclient: build_initial_monmap 2024-03-29T11:23:10.154 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.174+0000 7fbd3dc01640 10 monclient: monmap: 2024-03-29T11:23:10.154 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:23:10.154 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:23:10.154 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:23:10.175010+0000 2024-03-29T11:23:10.154 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:23:10.175010+0000 2024-03-29T11:23:10.154 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:10.154 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:23:10.154 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:23:10.154 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:23:10.154 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:23:10.154 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:23:10.155 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.177+0000 7fbd3dc01640 1 Processor -- start 2024-03-29T11:23:10.155 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.177+0000 7fbd3dc01640 1 -- start start 2024-03-29T11:23:10.155 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.177+0000 7fbd3dc01640 10 monclient: init 2024-03-29T11:23:10.156 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.178+0000 7fbd3dc01640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:10.156 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.178+0000 7fbd3dc01640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:23:10.156 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.178+0000 7fbd3dc01640 10 monclient(hunting): picked mon.noname-b con 0x7fbd380aaee0 addr v1:172.21.15.47:6790/0 2024-03-29T11:23:10.156 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.178+0000 7fbd3dc01640 10 monclient(hunting): picked mon.noname-c con 0x7fbd380aa500 addr v1:172.21.15.121:6789/0 2024-03-29T11:23:10.156 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.178+0000 7fbd3dc01640 10 monclient(hunting): picked mon.noname-a con 0x7fbd38149800 addr v1:172.21.15.47:6789/0 2024-03-29T11:23:10.156 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.178+0000 7fbd3dc01640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd380aded0 con 0x7fbd38149800 2024-03-29T11:23:10.156 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.178+0000 7fbd3dc01640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd380ad7e0 con 0x7fbd380aa500 2024-03-29T11:23:10.156 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.178+0000 7fbd3dc01640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd380ac350 con 0x7fbd380aaee0 2024-03-29T11:23:10.156 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.178+0000 7fbd3dc01640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:10.156 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.178+0000 7fbd3dc01640 10 monclient(hunting): authenticate will time out at 1433.366231s 2024-03-29T11:23:10.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.183+0000 7fbd3d400640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fbd38149800 0x7fbd38151ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:55561/0 (socket says 172.21.15.47:55561) 2024-03-29T11:23:10.161 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.183+0000 7fbd3d400640 1 -- 172.21.15.47:0/1749998673 learned_addr learned my addr 172.21.15.47:0/1749998673 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:23:10.166 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.187+0000 7fbd377fe640 1 -- 172.21.15.47:0/1749998673 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 253217699 0 0) 0x7fbd380ad7e0 con 0x7fbd380aa500 2024-03-29T11:23:10.166 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.187+0000 7fbd377fe640 10 monclient(hunting): _negotiate 2024-03-29T11:23:10.166 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.187+0000 7fbd377fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:10.166 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.187+0000 7fbd377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:10.166 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.187+0000 7fbd377fe640 10 monclient(hunting): my global_id is 4552 2024-03-29T11:23:10.166 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.187+0000 7fbd377fe640 1 -- 172.21.15.47:0/1749998673 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd20003520 con 0x7fbd380aa500 2024-03-29T11:23:10.166 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.187+0000 7fbd377fe640 1 -- 172.21.15.47:0/1749998673 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2259859738 0 0) 0x7fbd380aded0 con 0x7fbd38149800 2024-03-29T11:23:10.166 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.187+0000 7fbd377fe640 10 monclient(hunting): _negotiate 2024-03-29T11:23:10.166 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.187+0000 7fbd377fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:10.167 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.187+0000 7fbd377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:10.167 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.187+0000 7fbd377fe640 10 monclient(hunting): my global_id is 4581 2024-03-29T11:23:10.167 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.187+0000 7fbd377fe640 1 -- 172.21.15.47:0/1749998673 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd380ad7e0 con 0x7fbd38149800 2024-03-29T11:23:10.167 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.187+0000 7fbd377fe640 1 -- 172.21.15.47:0/1749998673 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2650857828 0 0) 0x7fbd20003520 con 0x7fbd380aa500 2024-03-29T11:23:10.167 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.187+0000 7fbd377fe640 1 -- 172.21.15.47:0/1749998673 >> v1:172.21.15.47:6790/0 conn(0x7fbd380aaee0 legacy=0x7fbd3814a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:23:10.167 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.187+0000 7fbd377fe640 1 -- 172.21.15.47:0/1749998673 >> v1:172.21.15.47:6789/0 conn(0x7fbd38149800 legacy=0x7fbd38151ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:10.167 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.187+0000 7fbd377fe640 10 monclient: _finish_hunting 0 2024-03-29T11:23:10.167 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.187+0000 7fbd377fe640 1 monclient: found mon.noname-c 2024-03-29T11:23:10.167 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.187+0000 7fbd377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:10.167 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.187+0000 7fbd377fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6789/0 2024-03-29T11:23:10.167 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.187+0000 7fbd377fe640 1 -- 172.21.15.47:0/1749998673 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd380ac630 con 0x7fbd380aa500 2024-03-29T11:23:10.167 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.187+0000 7fbd377fe640 10 monclient: _finish_auth 0 2024-03-29T11:23:10.167 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.188+0000 7fbd377fe640 10 monclient: _check_auth_tickets 2024-03-29T11:23:10.167 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.188+0000 7fbd377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:10.168 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.188+0000 7fbd3dc01640 5 monclient: authenticate success, global_id 4552 2024-03-29T11:23:10.168 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.188+0000 7fbd3dc01640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:10.168 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.188+0000 7fbd377fe640 1 -- 172.21.15.47:0/1749998673 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fbd28001690 con 0x7fbd380aa500 2024-03-29T11:23:10.168 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.188+0000 7fbd377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:10.168 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.188+0000 7fbd377fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:10.168 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.188+0000 7fbd377fe640 10 monclient: dump: 2024-03-29T11:23:10.168 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:23:10.168 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:23:10.168 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:23:10.168 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:23:10.168 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:10.168 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:23:10.168 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:23:10.168 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:23:10.169 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:23:10.169 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:23:10.169 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.188+0000 7fbd3dc01640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:10.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.188+0000 7fbd377fe640 1 -- 172.21.15.47:0/1749998673 <== mon.1 v1:172.21.15.121:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd280018b0 con 0x7fbd380aa500 2024-03-29T11:23:10.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.188+0000 7fbd377fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:10.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.188+0000 7fbd377fe640 1 -- 172.21.15.47:0/1749998673 <== mon.1 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fbd28001c40 con 0x7fbd380aa500 2024-03-29T11:23:10.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.188+0000 7fbd377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:10.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.188+0000 7fbd377fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:10.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.188+0000 7fbd377fe640 10 monclient: dump: 2024-03-29T11:23:10.171 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:23:10.171 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:23:10.171 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:23:10.171 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:23:10.171 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:10.171 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:23:10.171 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:23:10.171 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:23:10.171 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:23:10.171 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:23:10.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.188+0000 7fbd3dc01640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:10.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.188+0000 7fbd3dc01640 4 set_mon_vals no callback set 2024-03-29T11:23:10.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.189+0000 7fbd3dc01640 10 monclient: shutdown 2024-03-29T11:23:10.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.189+0000 7fbd3dc01640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:10.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.189+0000 7fbd3dc01640 1 -- 172.21.15.47:0/1749998673 >> v1:172.21.15.121:6789/0 conn(0x7fbd380aa500 legacy=0x7fbd380ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:10.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.189+0000 7fbd3dc01640 1 -- 172.21.15.47:0/1749998673 shutdown_connections 2024-03-29T11:23:10.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.189+0000 7fbd3dc01640 1 -- 172.21.15.47:0/1749998673 >> 172.21.15.47:0/1749998673 conn(0x7fbd380b3a30 msgr2=0x7fbd380b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:10.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.189+0000 7fbd3dc01640 1 -- 172.21.15.47:0/1749998673 shutdown_connections 2024-03-29T11:23:10.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.189+0000 7fbd3dc01640 1 -- 172.21.15.47:0/1749998673 wait complete. 2024-03-29T11:23:10.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.190+0000 7fbd3dc01640 10 monclient: build_initial_monmap 2024-03-29T11:23:10.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.190+0000 7fbd3dc01640 10 monclient: monmap: 2024-03-29T11:23:10.172 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:23:10.172 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:10.172 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:23:10.173 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:23:10.173 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:10.173 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:23:10.173 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:23:10.173 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:23:10.173 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:23:10.173 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:23:10.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.191+0000 7fbd3dc01640 1 librados: starting msgr at 2024-03-29T11:23:10.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.191+0000 7fbd3dc01640 1 librados: starting objecter 2024-03-29T11:23:10.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.191+0000 7fbd3dc01640 1 Processor -- start 2024-03-29T11:23:10.175 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.191+0000 7fbd3dc01640 1 -- start start 2024-03-29T11:23:10.175 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.191+0000 7fbd3dc01640 1 librados: setting wanted keys 2024-03-29T11:23:10.175 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.191+0000 7fbd3dc01640 1 librados: calling monclient init 2024-03-29T11:23:10.175 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.191+0000 7fbd3dc01640 10 monclient: init 2024-03-29T11:23:10.175 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.192+0000 7fbd3dc01640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:10.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.192+0000 7fbd3dc01640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:23:10.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.192+0000 7fbd3dc01640 10 monclient(hunting): picked mon.noname-c con 0x7fbd380aa500 addr v1:172.21.15.47:6790/0 2024-03-29T11:23:10.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.192+0000 7fbd3dc01640 10 monclient(hunting): picked mon.noname-b con 0x7fbd380aaee0 addr v1:172.21.15.121:6789/0 2024-03-29T11:23:10.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.192+0000 7fbd3dc01640 10 monclient(hunting): picked mon.noname-a con 0x7fbd38149800 addr v1:172.21.15.47:6789/0 2024-03-29T11:23:10.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.192+0000 7fbd3dc01640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd38145d70 con 0x7fbd38149800 2024-03-29T11:23:10.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.192+0000 7fbd3dc01640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd38146200 con 0x7fbd380aaee0 2024-03-29T11:23:10.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.192+0000 7fbd3dc01640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd38137c80 con 0x7fbd380aa500 2024-03-29T11:23:10.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.192+0000 7fbd3dc01640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:10.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.192+0000 7fbd3dc01640 10 monclient(hunting): authenticate will time out at 1433.380913s 2024-03-29T11:23:10.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.200+0000 7fbd3cbff640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fbd380aa500 0x7fbd38138490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:52467/0 (socket says 172.21.15.47:52467) 2024-03-29T11:23:10.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.200+0000 7fbd3cbff640 1 -- 172.21.15.47:0/3477664363 learned_addr learned my addr 172.21.15.47:0/3477664363 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:23:10.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.200+0000 7fbd37fff640 1 --1- 172.21.15.47:0/3477664363 >> v1:172.21.15.121:6789/0 conn(0x7fbd380aaee0 0x7fbd38136e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.47:51851/0 (socket says 172.21.15.47:51851) 2024-03-29T11:23:10.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.201+0000 7fbd357fa640 10 client.?.objecter ms_handle_connect 0x7fbd380aa500 2024-03-29T11:23:10.180 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.201+0000 7fbd357fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:10.180 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.201+0000 7fbd357fa640 10 client.?.objecter ms_handle_connect 0x7fbd380aaee0 2024-03-29T11:23:10.180 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.201+0000 7fbd357fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:10.180 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.201+0000 7fbd357fa640 1 -- 172.21.15.47:0/3477664363 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4025345562 0 0) 0x7fbd38137c80 con 0x7fbd380aa500 2024-03-29T11:23:10.180 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.201+0000 7fbd357fa640 10 monclient(hunting): _negotiate 2024-03-29T11:23:10.180 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.201+0000 7fbd357fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:10.180 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.201+0000 7fbd357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:10.181 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.202+0000 7fbd357fa640 10 monclient(hunting): my global_id is 4643 2024-03-29T11:23:10.181 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.202+0000 7fbd357fa640 1 -- 172.21.15.47:0/3477664363 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd24004eb0 con 0x7fbd380aa500 2024-03-29T11:23:10.181 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.202+0000 7fbd357fa640 10 client.?.objecter ms_handle_connect 0x7fbd38149800 2024-03-29T11:23:10.181 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.203+0000 7fbd357fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:10.182 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.203+0000 7fbd357fa640 1 -- 172.21.15.47:0/3477664363 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 625065567 0 0) 0x7fbd38146200 con 0x7fbd380aaee0 2024-03-29T11:23:10.182 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.203+0000 7fbd357fa640 10 monclient(hunting): _negotiate 2024-03-29T11:23:10.182 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.203+0000 7fbd357fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:10.182 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.203+0000 7fbd357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:10.182 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.203+0000 7fbd357fa640 10 monclient(hunting): my global_id is 4555 2024-03-29T11:23:10.182 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.203+0000 7fbd357fa640 1 -- 172.21.15.47:0/3477664363 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd38137c80 con 0x7fbd380aaee0 2024-03-29T11:23:10.183 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.204+0000 7fbd357fa640 1 -- 172.21.15.47:0/3477664363 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1877505552 0 0) 0x7fbd38145d70 con 0x7fbd38149800 2024-03-29T11:23:10.183 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.204+0000 7fbd357fa640 10 monclient(hunting): _negotiate 2024-03-29T11:23:10.183 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.204+0000 7fbd357fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:10.183 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.205+0000 7fbd357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:10.183 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.205+0000 7fbd357fa640 10 monclient(hunting): my global_id is 4584 2024-03-29T11:23:10.183 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.205+0000 7fbd357fa640 1 -- 172.21.15.47:0/3477664363 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd38146200 con 0x7fbd38149800 2024-03-29T11:23:10.183 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.205+0000 7fbd357fa640 1 -- 172.21.15.47:0/3477664363 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 585805739 0 0) 0x7fbd38137c80 con 0x7fbd380aaee0 2024-03-29T11:23:10.183 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.205+0000 7fbd357fa640 1 -- 172.21.15.47:0/3477664363 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd38145d70 con 0x7fbd380aaee0 2024-03-29T11:23:10.184 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.205+0000 7fbd357fa640 1 -- 172.21.15.47:0/3477664363 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fbd280015d0 con 0x7fbd380aaee0 2024-03-29T11:23:10.184 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.205+0000 7fbd357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:10.184 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.205+0000 7fbd357fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:10.184 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.205+0000 7fbd357fa640 10 monclient(hunting): dump: 2024-03-29T11:23:10.184 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:23:10.184 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:23:10.184 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:23:10.184 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:23:10.184 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:10.184 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:23:10.184 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:23:10.184 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:23:10.184 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:23:10.184 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:23:10.185 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.205+0000 7fbd357fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:10.186 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.205+0000 7fbd357fa640 1 -- 172.21.15.47:0/3477664363 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1993278188 0 0) 0x7fbd24004eb0 con 0x7fbd380aa500 2024-03-29T11:23:10.186 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.205+0000 7fbd357fa640 1 -- 172.21.15.47:0/3477664363 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd38137c80 con 0x7fbd380aa500 2024-03-29T11:23:10.186 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.205+0000 7fbd357fa640 1 -- 172.21.15.47:0/3477664363 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fbd30004400 con 0x7fbd380aa500 2024-03-29T11:23:10.186 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.205+0000 7fbd357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:10.186 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.205+0000 7fbd357fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:10.186 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.205+0000 7fbd357fa640 10 monclient(hunting): dump: 2024-03-29T11:23:10.186 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:23:10.186 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:23:10.186 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:23:10.186 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:23:10.186 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:10.187 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:23:10.187 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:23:10.187 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:23:10.187 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:23:10.187 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:23:10.187 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.205+0000 7fbd357fa640 1 -- 172.21.15.47:0/3477664363 <== mon.1 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1297352326 0 0) 0x7fbd38145d70 con 0x7fbd380aaee0 2024-03-29T11:23:10.187 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.206+0000 7fbd357fa640 1 -- 172.21.15.47:0/3477664363 >> v1:172.21.15.47:6790/0 conn(0x7fbd380aa500 legacy=0x7fbd38138490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:10.187 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.206+0000 7fbd357fa640 1 -- 172.21.15.47:0/3477664363 >> v1:172.21.15.47:6789/0 conn(0x7fbd38149800 legacy=0x7fbd38137570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:10.187 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.206+0000 7fbd357fa640 10 monclient: _finish_hunting 0 2024-03-29T11:23:10.187 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.206+0000 7fbd357fa640 1 monclient: found mon.b 2024-03-29T11:23:10.187 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.206+0000 7fbd357fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:10.187 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.206+0000 7fbd357fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:23:10.187 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.206+0000 7fbd357fa640 1 -- 172.21.15.47:0/3477664363 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd38147110 con 0x7fbd380aaee0 2024-03-29T11:23:10.187 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.206+0000 7fbd357fa640 10 monclient: _finish_auth 0 2024-03-29T11:23:10.187 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.206+0000 7fbd357fa640 10 monclient: _check_auth_tickets 2024-03-29T11:23:10.188 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.206+0000 7fbd357fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:10.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.206+0000 7fbd3dc01640 5 monclient: authenticate success, global_id 4555 2024-03-29T11:23:10.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.206+0000 7fbd3dc01640 10 monclient: _renew_subs 2024-03-29T11:23:10.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.206+0000 7fbd3dc01640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:23:10.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.206+0000 7fbd3dc01640 1 -- 172.21.15.47:0/3477664363 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd38144100 con 0x7fbd380aaee0 2024-03-29T11:23:10.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.206+0000 7fbd3dc01640 10 client.4555.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:10.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.206+0000 7fbd3dc01640 10 monclient: _renew_subs 2024-03-29T11:23:10.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.206+0000 7fbd3dc01640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:23:10.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.206+0000 7fbd3dc01640 1 -- 172.21.15.47:0/3477664363 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd38145170 con 0x7fbd380aaee0 2024-03-29T11:23:10.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.206+0000 7fbd3dc01640 1 librados: init done 2024-03-29T11:23:10.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.212+0000 7fbd357fa640 1 -- 172.21.15.47:0/3477664363 <== mon.1 v1:172.21.15.121:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd280038a0 con 0x7fbd380aaee0 2024-03-29T11:23:10.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.212+0000 7fbd357fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:10.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.212+0000 7fbd357fa640 1 -- 172.21.15.47:0/3477664363 <== mon.1 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fbd28003c30 con 0x7fbd380aaee0 2024-03-29T11:23:10.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.212+0000 7fbd357fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:10.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.212+0000 7fbd357fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:10.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.212+0000 7fbd357fa640 10 monclient: dump: 2024-03-29T11:23:10.193 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:23:10.193 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:23:10.193 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:23:10.193 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:23:10.193 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:10.193 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:23:10.193 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:23:10.193 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:23:10.193 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:23:10.193 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:23:10.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.212+0000 7fbd357fa640 1 -- 172.21.15.47:0/3477664363 <== mon.1 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7fbd28003e50 con 0x7fbd380aaee0 2024-03-29T11:23:10.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.213+0000 7fbd357fa640 1 -- 172.21.15.47:0/3477664363 <== mon.1 v1:172.21.15.121:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3418925575 0 0) 0x7fbd28084b90 con 0x7fbd380aaee0 2024-03-29T11:23:10.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.213+0000 7fbd357fa640 10 client.4555.objecter ms_dispatch 0x7fbd38000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:23:10.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.213+0000 7fbd357fa640 3 client.4555.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:23:10.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.213+0000 7fbd357fa640 3 client.4555.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:23:10.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.213+0000 7fbd357fa640 20 client.4555.objecter dump_active .. 0 homeless 2024-03-29T11:23:10.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.213+0000 7fbd36ffd640 4 set_mon_vals no callback set 2024-03-29T11:23:10.195 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.214+0000 7fbd3cbff640 10 monclient: get_auth_request con 0x7fbd2406dd70 auth_method 0 2024-03-29T11:23:10.195 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.214+0000 7fbd3dc01640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:10.195 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.215+0000 7fbd357fa640 10 client.4555.objecter ms_handle_connect 0x7fbd2406dd70 2024-03-29T11:23:10.195 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.215+0000 7fbd3dc01640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:10.195 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.215+0000 7fbd3dc01640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:23:10.195 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.215+0000 7fbd3dc01640 1 -- 172.21.15.47:0/3477664363 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd18005080 con 0x7fbd380aaee0 2024-03-29T11:23:10.202 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.224+0000 7fbd357fa640 1 -- 172.21.15.47:0/3477664363 <== mon.1 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbd280504f0 con 0x7fbd380aaee0 2024-03-29T11:23:10.202 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.224+0000 7fbd357fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:10.202 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.224+0000 7fbd357fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:10.558 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.579+0000 7fbd3dc01640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:10.558 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.580+0000 7fbd3dc01640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:10.558 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.580+0000 7fbd3dc01640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6789/0 2024-03-29T11:23:10.558 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.580+0000 7fbd3dc01640 1 -- 172.21.15.47:0/3477664363 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbd18005330 con 0x7fbd380aaee0 2024-03-29T11:23:10.559 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.580+0000 7fbd357fa640 1 -- 172.21.15.47:0/3477664363 <== mon.1 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1178522771) 0x7fbd280504f0 con 0x7fbd380aaee0 2024-03-29T11:23:10.559 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.580+0000 7fbd357fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:10.559 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.580+0000 7fbd357fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:10.559 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:23:10.559 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":11,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:22:45.925385+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:22:45.925385+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:10.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.584+0000 7fbd3dc01640 10 librados: watch_flush enter 2024-03-29T11:23:10.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.586+0000 7fbd3dc01640 10 librados: watch_flush exit 2024-03-29T11:23:10.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.586+0000 7fbd3dc01640 20 client.4555.objecter shutdown clearing up homeless session... 2024-03-29T11:23:10.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.586+0000 7fbd3dc01640 10 client.4555.objecter successfully canceled tick 2024-03-29T11:23:10.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.586+0000 7fbd3dc01640 1 -- 172.21.15.47:0/3477664363 >> v1:172.21.15.121:6816/1964117370 conn(0x7fbd2406dd70 legacy=0x7fbd24070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:10.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.586+0000 7fbd3dc01640 10 monclient: shutdown 2024-03-29T11:23:10.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.586+0000 7fbd3dc01640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:10.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.586+0000 7fbd3dc01640 1 -- 172.21.15.47:0/3477664363 >> v1:172.21.15.121:6789/0 conn(0x7fbd380aaee0 legacy=0x7fbd38136e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:10.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.586+0000 7fbd3dc01640 1 -- 172.21.15.47:0/3477664363 shutdown_connections 2024-03-29T11:23:10.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.586+0000 7fbd3dc01640 1 -- 172.21.15.47:0/3477664363 >> 172.21.15.47:0/3477664363 conn(0x7fbd380b3a30 msgr2=0x7fbd3813d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:10.566 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.587+0000 7fbd3dc01640 1 -- 172.21.15.47:0/3477664363 shutdown_connections 2024-03-29T11:23:10.566 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.587+0000 7fbd3dc01640 1 -- 172.21.15.47:0/3477664363 wait complete. 2024-03-29T11:23:10.566 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:10.588+0000 7fbd3dc01640 1 librados: shutdown 2024-03-29T11:23:10.587 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:17.394 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:17.542 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.563+0000 7fdb815af640 10 monclient: get_monmap_and_config 2024-03-29T11:23:17.543 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.563+0000 7fdb815af640 10 monclient: build_initial_monmap 2024-03-29T11:23:17.543 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.564+0000 7fdb815af640 10 monclient: monmap: 2024-03-29T11:23:17.543 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:23:17.543 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:23:17.543 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:23:17.564941+0000 2024-03-29T11:23:17.543 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:23:17.564941+0000 2024-03-29T11:23:17.543 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:17.543 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:23:17.543 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:23:17.543 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:23:17.544 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:23:17.544 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:23:17.547 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.564+0000 7fdb815af640 1 Processor -- start 2024-03-29T11:23:17.547 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.564+0000 7fdb815af640 1 -- start start 2024-03-29T11:23:17.547 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.565+0000 7fdb815af640 10 monclient: init 2024-03-29T11:23:17.547 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.565+0000 7fdb815af640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:17.547 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.565+0000 7fdb815af640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:23:17.547 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.565+0000 7fdb815af640 10 monclient(hunting): picked mon.noname-a con 0x7fdb7c0b1f60 addr v1:172.21.15.47:6789/0 2024-03-29T11:23:17.547 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.565+0000 7fdb815af640 10 monclient(hunting): picked mon.noname-c con 0x7fdb7c071eb0 addr v1:172.21.15.121:6789/0 2024-03-29T11:23:17.547 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.565+0000 7fdb815af640 10 monclient(hunting): picked mon.noname-b con 0x7fdb7c143d10 addr v1:172.21.15.47:6790/0 2024-03-29T11:23:17.548 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.565+0000 7fdb815af640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb7c06ef80 con 0x7fdb7c0b1f60 2024-03-29T11:23:17.548 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.565+0000 7fdb815af640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb7c06f260 con 0x7fdb7c071eb0 2024-03-29T11:23:17.548 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.565+0000 7fdb815af640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb7c0b40c0 con 0x7fdb7c143d10 2024-03-29T11:23:17.548 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.565+0000 7fdb815af640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:17.548 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.565+0000 7fdb815af640 10 monclient(hunting): authenticate will time out at 1440.753852s 2024-03-29T11:23:17.548 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.566+0000 7fdb7b7fe640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fdb7c143d10 0x7fdb7c146b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:51148/0 (socket says 172.21.15.47:51148) 2024-03-29T11:23:17.548 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.566+0000 7fdb7b7fe640 1 -- 172.21.15.47:0/2690219770 learned_addr learned my addr 172.21.15.47:0/2690219770 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:23:17.548 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.567+0000 7fdb79ffb640 1 -- 172.21.15.47:0/2690219770 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 785859394 0 0) 0x7fdb7c06ef80 con 0x7fdb7c0b1f60 2024-03-29T11:23:17.548 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.567+0000 7fdb79ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:17.548 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.567+0000 7fdb79ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:17.548 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.567+0000 7fdb79ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:17.548 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.567+0000 7fdb79ffb640 10 monclient(hunting): my global_id is 4587 2024-03-29T11:23:17.548 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.567+0000 7fdb79ffb640 1 -- 172.21.15.47:0/2690219770 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb64003520 con 0x7fdb7c0b1f60 2024-03-29T11:23:17.548 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.567+0000 7fdb79ffb640 1 -- 172.21.15.47:0/2690219770 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1034693645 0 0) 0x7fdb7c0b40c0 con 0x7fdb7c143d10 2024-03-29T11:23:17.548 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.567+0000 7fdb79ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:17.548 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.567+0000 7fdb79ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:17.549 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.567+0000 7fdb79ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:17.549 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.567+0000 7fdb79ffb640 10 monclient(hunting): my global_id is 4646 2024-03-29T11:23:17.549 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.567+0000 7fdb79ffb640 1 -- 172.21.15.47:0/2690219770 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb7c06ef80 con 0x7fdb7c143d10 2024-03-29T11:23:17.549 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.568+0000 7fdb79ffb640 1 -- 172.21.15.47:0/2690219770 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3964564665 0 0) 0x7fdb64003520 con 0x7fdb7c0b1f60 2024-03-29T11:23:17.549 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.568+0000 7fdb79ffb640 1 -- 172.21.15.47:0/2690219770 >> v1:172.21.15.47:6790/0 conn(0x7fdb7c143d10 legacy=0x7fdb7c146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:17.549 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.568+0000 7fdb79ffb640 1 -- 172.21.15.47:0/2690219770 >> v1:172.21.15.121:6789/0 conn(0x7fdb7c071eb0 legacy=0x7fdb7c146470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:23:17.549 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.568+0000 7fdb79ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:23:17.549 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.568+0000 7fdb79ffb640 1 monclient: found mon.noname-a 2024-03-29T11:23:17.549 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.569+0000 7fdb79ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:17.549 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.569+0000 7fdb79ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-29T11:23:17.549 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.569+0000 7fdb79ffb640 1 -- 172.21.15.47:0/2690219770 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb7c155d70 con 0x7fdb7c0b1f60 2024-03-29T11:23:17.549 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.569+0000 7fdb79ffb640 10 monclient: _finish_auth 0 2024-03-29T11:23:17.549 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.569+0000 7fdb79ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:23:17.550 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.569+0000 7fdb79ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:17.550 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.569+0000 7fdb79ffb640 1 -- 172.21.15.47:0/2690219770 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fdb6c0015c0 con 0x7fdb7c0b1f60 2024-03-29T11:23:17.550 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.569+0000 7fdb79ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:17.550 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.569+0000 7fdb79ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:23:17.550 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.569+0000 7fdb79ffb640 10 monclient: dump: 2024-03-29T11:23:17.550 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:23:17.550 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:23:17.550 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:23:17.550 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:23:17.550 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:17.550 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:23:17.550 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:23:17.550 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:23:17.551 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:23:17.551 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:23:17.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.569+0000 7fdb815af640 5 monclient: authenticate success, global_id 4587 2024-03-29T11:23:17.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.569+0000 7fdb815af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:17.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.569+0000 7fdb79ffb640 1 -- 172.21.15.47:0/2690219770 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb6c001720 con 0x7fdb7c0b1f60 2024-03-29T11:23:17.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.569+0000 7fdb79ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:17.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.569+0000 7fdb815af640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:17.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.569+0000 7fdb815af640 4 set_mon_vals no callback set 2024-03-29T11:23:17.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.570+0000 7fdb79ffb640 1 -- 172.21.15.47:0/2690219770 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fdb6c001b70 con 0x7fdb7c0b1f60 2024-03-29T11:23:17.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.570+0000 7fdb79ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:17.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.570+0000 7fdb79ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:17.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.570+0000 7fdb79ffb640 10 monclient: dump: 2024-03-29T11:23:17.552 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:23:17.552 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:23:17.552 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:23:17.553 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:23:17.553 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:17.553 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:23:17.553 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:23:17.553 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:23:17.553 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:23:17.553 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:23:17.553 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.570+0000 7fdb815af640 10 monclient: shutdown 2024-03-29T11:23:17.553 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.570+0000 7fdb815af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:17.553 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.570+0000 7fdb815af640 1 -- 172.21.15.47:0/2690219770 >> v1:172.21.15.47:6789/0 conn(0x7fdb7c0b1f60 legacy=0x7fdb7c143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:17.553 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.570+0000 7fdb815af640 1 -- 172.21.15.47:0/2690219770 shutdown_connections 2024-03-29T11:23:17.553 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.570+0000 7fdb815af640 1 -- 172.21.15.47:0/2690219770 >> 172.21.15.47:0/2690219770 conn(0x7fdb7c066a00 msgr2=0x7fdb7c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:17.553 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.570+0000 7fdb815af640 1 -- 172.21.15.47:0/2690219770 shutdown_connections 2024-03-29T11:23:17.553 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.571+0000 7fdb815af640 1 -- 172.21.15.47:0/2690219770 wait complete. 2024-03-29T11:23:17.554 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.571+0000 7fdb815af640 10 monclient: build_initial_monmap 2024-03-29T11:23:17.554 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.571+0000 7fdb815af640 10 monclient: monmap: 2024-03-29T11:23:17.554 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:23:17.554 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:17.554 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:23:17.554 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:23:17.554 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:17.554 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:23:17.554 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:23:17.554 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:23:17.554 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:23:17.554 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:23:17.554 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.572+0000 7fdb815af640 1 librados: starting msgr at 2024-03-29T11:23:17.555 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.572+0000 7fdb815af640 1 librados: starting objecter 2024-03-29T11:23:17.555 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.572+0000 7fdb815af640 1 Processor -- start 2024-03-29T11:23:17.555 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.572+0000 7fdb815af640 1 -- start start 2024-03-29T11:23:17.555 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.572+0000 7fdb815af640 1 librados: setting wanted keys 2024-03-29T11:23:17.555 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.572+0000 7fdb815af640 1 librados: calling monclient init 2024-03-29T11:23:17.555 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.572+0000 7fdb815af640 10 monclient: init 2024-03-29T11:23:17.557 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.573+0000 7fdb815af640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:17.557 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.573+0000 7fdb815af640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:23:17.557 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.573+0000 7fdb815af640 10 monclient(hunting): picked mon.noname-c con 0x7fdb7c0b1f60 addr v1:172.21.15.47:6790/0 2024-03-29T11:23:17.557 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.573+0000 7fdb815af640 10 monclient(hunting): picked mon.noname-a con 0x7fdb7c143d10 addr v1:172.21.15.47:6789/0 2024-03-29T11:23:17.557 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.573+0000 7fdb815af640 10 monclient(hunting): picked mon.noname-b con 0x7fdb7c1552d0 addr v1:172.21.15.121:6789/0 2024-03-29T11:23:17.557 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.573+0000 7fdb815af640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb7c142cb0 con 0x7fdb7c143d10 2024-03-29T11:23:17.557 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.573+0000 7fdb815af640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb7c070670 con 0x7fdb7c1552d0 2024-03-29T11:23:17.558 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.573+0000 7fdb815af640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb7c070950 con 0x7fdb7c0b1f60 2024-03-29T11:23:17.558 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.573+0000 7fdb815af640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:17.558 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.573+0000 7fdb815af640 10 monclient(hunting): authenticate will time out at 1440.761570s 2024-03-29T11:23:17.558 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.573+0000 7fdb7affd640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fdb7c0b1f60 0x7fdb7c071050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:51988/0 (socket says 172.21.15.47:51988) 2024-03-29T11:23:17.558 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.573+0000 7fdb7affd640 1 -- 172.21.15.47:0/4287693843 learned_addr learned my addr 172.21.15.47:0/4287693843 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:23:17.558 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.573+0000 7fdb637fe640 10 client.?.objecter ms_handle_connect 0x7fdb7c0b1f60 2024-03-29T11:23:17.558 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.573+0000 7fdb637fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:17.558 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.574+0000 7fdb637fe640 1 -- 172.21.15.47:0/4287693843 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 377935513 0 0) 0x7fdb7c070950 con 0x7fdb7c0b1f60 2024-03-29T11:23:17.558 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.574+0000 7fdb637fe640 10 monclient(hunting): _negotiate 2024-03-29T11:23:17.558 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.574+0000 7fdb637fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:17.558 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.574+0000 7fdb637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:17.558 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.574+0000 7fdb637fe640 10 monclient(hunting): my global_id is 4649 2024-03-29T11:23:17.558 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.574+0000 7fdb637fe640 1 -- 172.21.15.47:0/4287693843 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb68004eb0 con 0x7fdb7c0b1f60 2024-03-29T11:23:17.558 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.574+0000 7fdb637fe640 10 client.?.objecter ms_handle_connect 0x7fdb7c143d10 2024-03-29T11:23:17.559 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.574+0000 7fdb637fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:17.559 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.574+0000 7fdb637fe640 1 -- 172.21.15.47:0/4287693843 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3324190871 0 0) 0x7fdb7c142cb0 con 0x7fdb7c143d10 2024-03-29T11:23:17.559 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.574+0000 7fdb637fe640 10 monclient(hunting): _negotiate 2024-03-29T11:23:17.559 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.574+0000 7fdb637fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:17.559 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.574+0000 7fdb637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:17.559 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.574+0000 7fdb637fe640 10 monclient(hunting): my global_id is 4590 2024-03-29T11:23:17.559 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.574+0000 7fdb637fe640 1 -- 172.21.15.47:0/4287693843 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb7c070950 con 0x7fdb7c143d10 2024-03-29T11:23:17.559 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.574+0000 7fdb637fe640 1 -- 172.21.15.47:0/4287693843 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1441271804 0 0) 0x7fdb68004eb0 con 0x7fdb7c0b1f60 2024-03-29T11:23:17.559 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.574+0000 7fdb637fe640 1 -- 172.21.15.47:0/4287693843 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb7c142cb0 con 0x7fdb7c0b1f60 2024-03-29T11:23:17.559 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.575+0000 7fdb637fe640 1 -- 172.21.15.47:0/4287693843 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fdb6c001b30 con 0x7fdb7c0b1f60 2024-03-29T11:23:17.559 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.575+0000 7fdb637fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:17.559 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.575+0000 7fdb637fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:17.559 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.575+0000 7fdb637fe640 10 monclient(hunting): dump: 2024-03-29T11:23:17.559 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:23:17.559 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:23:17.560 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:23:17.560 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:23:17.560 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:17.560 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:23:17.560 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:23:17.560 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:23:17.560 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:23:17.560 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:23:17.560 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.575+0000 7fdb637fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:17.560 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.575+0000 7fdb637fe640 1 -- 172.21.15.47:0/4287693843 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2283264045 0 0) 0x7fdb7c142cb0 con 0x7fdb7c0b1f60 2024-03-29T11:23:17.560 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.575+0000 7fdb637fe640 1 -- 172.21.15.47:0/4287693843 >> v1:172.21.15.121:6789/0 conn(0x7fdb7c1552d0 legacy=0x7fdb7c131b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:23:17.560 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.577+0000 7fdb637fe640 1 -- 172.21.15.47:0/4287693843 >> v1:172.21.15.47:6789/0 conn(0x7fdb7c143d10 legacy=0x7fdb7c154540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:17.560 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.577+0000 7fdb637fe640 10 monclient: _finish_hunting 0 2024-03-29T11:23:17.560 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.577+0000 7fdb637fe640 1 monclient: found mon.c 2024-03-29T11:23:17.561 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.577+0000 7fdb637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:17.561 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.577+0000 7fdb637fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:23:17.561 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.577+0000 7fdb637fe640 1 -- 172.21.15.47:0/4287693843 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb7c133180 con 0x7fdb7c0b1f60 2024-03-29T11:23:17.561 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.577+0000 7fdb637fe640 10 monclient: _finish_auth 0 2024-03-29T11:23:17.561 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.577+0000 7fdb637fe640 10 monclient: _check_auth_tickets 2024-03-29T11:23:17.561 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.577+0000 7fdb637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:17.561 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.578+0000 7fdb637fe640 1 -- 172.21.15.47:0/4287693843 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb6c0038a0 con 0x7fdb7c0b1f60 2024-03-29T11:23:17.561 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.578+0000 7fdb637fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:17.563 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.578+0000 7fdb797fa640 4 set_mon_vals no callback set 2024-03-29T11:23:17.563 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.579+0000 7fdb637fe640 1 -- 172.21.15.47:0/4287693843 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fdb6c003c30 con 0x7fdb7c0b1f60 2024-03-29T11:23:17.563 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.579+0000 7fdb637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:17.563 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.579+0000 7fdb637fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:17.563 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.579+0000 7fdb637fe640 10 monclient: dump: 2024-03-29T11:23:17.563 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:23:17.563 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:23:17.563 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:23:17.563 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:23:17.564 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:17.564 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:23:17.564 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:23:17.564 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:23:17.564 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:23:17.564 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:23:17.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.579+0000 7fdb815af640 5 monclient: authenticate success, global_id 4649 2024-03-29T11:23:17.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.579+0000 7fdb815af640 10 monclient: _renew_subs 2024-03-29T11:23:17.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.579+0000 7fdb815af640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:23:17.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.579+0000 7fdb815af640 1 -- 172.21.15.47:0/4287693843 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb7c06f7f0 con 0x7fdb7c0b1f60 2024-03-29T11:23:17.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.579+0000 7fdb815af640 10 client.4649.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:17.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.579+0000 7fdb815af640 10 monclient: _renew_subs 2024-03-29T11:23:17.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.579+0000 7fdb815af640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:23:17.564 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.579+0000 7fdb815af640 1 -- 172.21.15.47:0/4287693843 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb7c1322a0 con 0x7fdb7c0b1f60 2024-03-29T11:23:17.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.579+0000 7fdb815af640 1 librados: init done 2024-03-29T11:23:17.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.581+0000 7fdb815af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:17.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.581+0000 7fdb815af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:17.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.581+0000 7fdb815af640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:23:17.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.581+0000 7fdb815af640 1 -- 172.21.15.47:0/4287693843 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb7c075f10 con 0x7fdb7c0b1f60 2024-03-29T11:23:17.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.581+0000 7fdb637fe640 1 -- 172.21.15.47:0/4287693843 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7fdb6c003d90 con 0x7fdb7c0b1f60 2024-03-29T11:23:17.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.582+0000 7fdb637fe640 1 -- 172.21.15.47:0/4287693843 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3418925575 0 0) 0x7fdb6c084b30 con 0x7fdb7c0b1f60 2024-03-29T11:23:17.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.582+0000 7fdb7a7fc640 10 monclient: get_auth_request con 0x7fdb6806da00 auth_method 0 2024-03-29T11:23:17.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.582+0000 7fdb637fe640 10 client.4649.objecter ms_dispatch 0x7fdb7c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:23:17.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.582+0000 7fdb637fe640 3 client.4649.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:23:17.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.582+0000 7fdb637fe640 3 client.4649.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:23:17.565 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.582+0000 7fdb637fe640 20 client.4649.objecter dump_active .. 0 homeless 2024-03-29T11:23:17.566 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.582+0000 7fdb637fe640 10 client.4649.objecter ms_handle_connect 0x7fdb6806da00 2024-03-29T11:23:17.569 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.591+0000 7fdb637fe640 1 -- 172.21.15.47:0/4287693843 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdb6c050470 con 0x7fdb7c0b1f60 2024-03-29T11:23:17.569 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.591+0000 7fdb637fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:17.569 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.591+0000 7fdb637fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:17.903 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.924+0000 7fdb815af640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:17.903 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.924+0000 7fdb815af640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:17.903 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.924+0000 7fdb815af640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:23:17.903 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.924+0000 7fdb815af640 1 -- 172.21.15.47:0/4287693843 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdb7c143050 con 0x7fdb7c0b1f60 2024-03-29T11:23:17.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.927+0000 7fdb637fe640 1 -- 172.21.15.47:0/4287693843 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1178522771) 0x7fdb6c0841e0 con 0x7fdb7c0b1f60 2024-03-29T11:23:17.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.928+0000 7fdb637fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:17.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.928+0000 7fdb637fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:17.907 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:23:17.907 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":11,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:22:45.925385+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:22:45.925385+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:17.910 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.931+0000 7fdb815af640 10 librados: watch_flush enter 2024-03-29T11:23:17.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.933+0000 7fdb815af640 10 librados: watch_flush exit 2024-03-29T11:23:17.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.933+0000 7fdb815af640 20 client.4649.objecter shutdown clearing up homeless session... 2024-03-29T11:23:17.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.933+0000 7fdb815af640 10 client.4649.objecter successfully canceled tick 2024-03-29T11:23:17.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.933+0000 7fdb815af640 1 -- 172.21.15.47:0/4287693843 >> v1:172.21.15.121:6816/1964117370 conn(0x7fdb6806da00 legacy=0x7fdb6806fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:17.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.933+0000 7fdb815af640 10 monclient: shutdown 2024-03-29T11:23:17.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.933+0000 7fdb815af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:17.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.933+0000 7fdb815af640 1 -- 172.21.15.47:0/4287693843 >> v1:172.21.15.47:6790/0 conn(0x7fdb7c0b1f60 legacy=0x7fdb7c071050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:17.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.933+0000 7fdb815af640 1 -- 172.21.15.47:0/4287693843 shutdown_connections 2024-03-29T11:23:17.913 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.933+0000 7fdb815af640 1 -- 172.21.15.47:0/4287693843 >> 172.21.15.47:0/4287693843 conn(0x7fdb7c066a00 msgr2=0x7fdb7c0b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:17.913 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.933+0000 7fdb815af640 1 -- 172.21.15.47:0/4287693843 shutdown_connections 2024-03-29T11:23:17.913 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.934+0000 7fdb815af640 1 -- 172.21.15.47:0/4287693843 wait complete. 2024-03-29T11:23:17.913 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:17.934+0000 7fdb815af640 1 librados: shutdown 2024-03-29T11:23:17.931 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:24.738 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:24.886 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.907+0000 7fec6b04d640 10 monclient: get_monmap_and_config 2024-03-29T11:23:24.886 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.907+0000 7fec6b04d640 10 monclient: build_initial_monmap 2024-03-29T11:23:24.886 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.907+0000 7fec6b04d640 10 monclient: monmap: 2024-03-29T11:23:24.886 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:23:24.886 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:23:24.886 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:23:24.908147+0000 2024-03-29T11:23:24.886 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:23:24.908147+0000 2024-03-29T11:23:24.886 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:24.886 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:23:24.886 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:23:24.887 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:23:24.887 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:23:24.887 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:23:24.887 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.908+0000 7fec6b04d640 1 Processor -- start 2024-03-29T11:23:24.887 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.908+0000 7fec6b04d640 1 -- start start 2024-03-29T11:23:24.887 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.908+0000 7fec6b04d640 10 monclient: init 2024-03-29T11:23:24.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.909+0000 7fec6b04d640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:24.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.909+0000 7fec6b04d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:23:24.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.909+0000 7fec6b04d640 10 monclient(hunting): picked mon.noname-c con 0x7fec641422b0 addr v1:172.21.15.121:6789/0 2024-03-29T11:23:24.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.909+0000 7fec6b04d640 10 monclient(hunting): picked mon.noname-b con 0x7fec64142690 addr v1:172.21.15.47:6790/0 2024-03-29T11:23:24.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.909+0000 7fec6b04d640 10 monclient(hunting): picked mon.noname-a con 0x7fec640b17d0 addr v1:172.21.15.47:6789/0 2024-03-29T11:23:24.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.909+0000 7fec6b04d640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec64142c80 con 0x7fec640b17d0 2024-03-29T11:23:24.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.909+0000 7fec6b04d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec640aae40 con 0x7fec641422b0 2024-03-29T11:23:24.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.909+0000 7fec6b04d640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec6413fbd0 con 0x7fec64142690 2024-03-29T11:23:24.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.909+0000 7fec6b04d640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:24.888 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.909+0000 7fec6b04d640 10 monclient(hunting): authenticate will time out at 1448.097402s 2024-03-29T11:23:24.889 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.909+0000 7fec695ea640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fec640b17d0 0x7fec640b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:48482/0 (socket says 172.21.15.47:48482) 2024-03-29T11:23:24.889 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.909+0000 7fec695ea640 1 -- 172.21.15.47:0/2601659059 learned_addr learned my addr 172.21.15.47:0/2601659059 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:23:24.889 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.910+0000 7fec637fe640 1 -- 172.21.15.47:0/2601659059 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 858923062 0 0) 0x7fec64142c80 con 0x7fec640b17d0 2024-03-29T11:23:24.890 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.910+0000 7fec637fe640 10 monclient(hunting): _negotiate 2024-03-29T11:23:24.890 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.910+0000 7fec637fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:24.890 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.910+0000 7fec637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:24.890 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.910+0000 7fec637fe640 10 monclient(hunting): my global_id is 4593 2024-03-29T11:23:24.891 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.910+0000 7fec637fe640 1 -- 172.21.15.47:0/2601659059 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec54003520 con 0x7fec640b17d0 2024-03-29T11:23:24.891 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.910+0000 7fec637fe640 1 -- 172.21.15.47:0/2601659059 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1933340153 0 0) 0x7fec54003520 con 0x7fec640b17d0 2024-03-29T11:23:24.891 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.910+0000 7fec637fe640 1 -- 172.21.15.47:0/2601659059 >> v1:172.21.15.47:6790/0 conn(0x7fec64142690 legacy=0x7fec640b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:24.891 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.910+0000 7fec637fe640 1 -- 172.21.15.47:0/2601659059 >> v1:172.21.15.121:6789/0 conn(0x7fec641422b0 legacy=0x7fec6413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:24.891 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.910+0000 7fec637fe640 10 monclient: _finish_hunting 0 2024-03-29T11:23:24.891 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.910+0000 7fec637fe640 1 monclient: found mon.noname-a 2024-03-29T11:23:24.891 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.910+0000 7fec637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:24.891 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.910+0000 7fec637fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-29T11:23:24.891 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.910+0000 7fec637fe640 1 -- 172.21.15.47:0/2601659059 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec64151940 con 0x7fec640b17d0 2024-03-29T11:23:24.891 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.910+0000 7fec637fe640 10 monclient: _finish_auth 0 2024-03-29T11:23:24.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.910+0000 7fec637fe640 10 monclient: _check_auth_tickets 2024-03-29T11:23:24.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.910+0000 7fec637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:24.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.910+0000 7fec6b04d640 5 monclient: authenticate success, global_id 4593 2024-03-29T11:23:24.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.910+0000 7fec6b04d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:24.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.910+0000 7fec637fe640 1 -- 172.21.15.47:0/2601659059 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fec5c001a10 con 0x7fec640b17d0 2024-03-29T11:23:24.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.910+0000 7fec637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:24.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.910+0000 7fec637fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:23:24.892 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.910+0000 7fec637fe640 10 monclient: dump: 2024-03-29T11:23:24.892 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:23:24.892 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:23:24.892 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:23:24.892 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:23:24.892 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:24.892 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:23:24.892 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:23:24.893 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:23:24.893 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:23:24.893 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:23:24.893 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.910+0000 7fec6b04d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:24.893 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.910+0000 7fec637fe640 1 -- 172.21.15.47:0/2601659059 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec5c001c30 con 0x7fec640b17d0 2024-03-29T11:23:24.893 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.910+0000 7fec637fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:24.893 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.910+0000 7fec6b04d640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:24.893 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.910+0000 7fec637fe640 1 -- 172.21.15.47:0/2601659059 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fec5c001fc0 con 0x7fec640b17d0 2024-03-29T11:23:24.893 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.910+0000 7fec6b04d640 4 set_mon_vals no callback set 2024-03-29T11:23:24.893 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.910+0000 7fec637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:24.893 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.910+0000 7fec637fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:24.893 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.910+0000 7fec637fe640 10 monclient: dump: 2024-03-29T11:23:24.893 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:23:24.893 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:23:24.893 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:23:24.894 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:23:24.894 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:24.894 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:23:24.894 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:23:24.894 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:23:24.894 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:23:24.894 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:23:24.903 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.911+0000 7fec6b04d640 10 monclient: shutdown 2024-03-29T11:23:24.903 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.911+0000 7fec6b04d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:24.903 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.911+0000 7fec6b04d640 1 -- 172.21.15.47:0/2601659059 >> v1:172.21.15.47:6789/0 conn(0x7fec640b17d0 legacy=0x7fec640b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:24.903 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.911+0000 7fec6b04d640 1 -- 172.21.15.47:0/2601659059 shutdown_connections 2024-03-29T11:23:24.903 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.911+0000 7fec6b04d640 1 -- 172.21.15.47:0/2601659059 >> 172.21.15.47:0/2601659059 conn(0x7fec64066a00 msgr2=0x7fec64066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:24.903 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.911+0000 7fec6b04d640 1 -- 172.21.15.47:0/2601659059 shutdown_connections 2024-03-29T11:23:24.904 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.911+0000 7fec6b04d640 1 -- 172.21.15.47:0/2601659059 wait complete. 2024-03-29T11:23:24.904 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.912+0000 7fec6b04d640 10 monclient: build_initial_monmap 2024-03-29T11:23:24.904 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.912+0000 7fec6b04d640 10 monclient: monmap: 2024-03-29T11:23:24.904 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:23:24.904 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:24.904 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:23:24.904 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:23:24.904 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:24.904 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:23:24.904 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:23:24.904 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:23:24.904 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:23:24.904 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:23:24.904 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.913+0000 7fec6b04d640 1 librados: starting msgr at 2024-03-29T11:23:24.904 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.913+0000 7fec6b04d640 1 librados: starting objecter 2024-03-29T11:23:24.904 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.913+0000 7fec6b04d640 1 Processor -- start 2024-03-29T11:23:24.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.913+0000 7fec6b04d640 1 -- start start 2024-03-29T11:23:24.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.913+0000 7fec6b04d640 1 librados: setting wanted keys 2024-03-29T11:23:24.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.913+0000 7fec6b04d640 1 librados: calling monclient init 2024-03-29T11:23:24.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.913+0000 7fec6b04d640 10 monclient: init 2024-03-29T11:23:24.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.914+0000 7fec6b04d640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:24.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.914+0000 7fec6b04d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:23:24.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.914+0000 7fec6b04d640 10 monclient(hunting): picked mon.noname-a con 0x7fec640b17d0 addr v1:172.21.15.47:6789/0 2024-03-29T11:23:24.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.914+0000 7fec6b04d640 10 monclient(hunting): picked mon.noname-b con 0x7fec641422b0 addr v1:172.21.15.121:6789/0 2024-03-29T11:23:24.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.914+0000 7fec6b04d640 10 monclient(hunting): picked mon.noname-c con 0x7fec64142690 addr v1:172.21.15.47:6790/0 2024-03-29T11:23:24.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.914+0000 7fec6b04d640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec640ac4e0 con 0x7fec640b17d0 2024-03-29T11:23:24.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.914+0000 7fec6b04d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec640b11f0 con 0x7fec641422b0 2024-03-29T11:23:24.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.914+0000 7fec63fff640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fec641422b0 0x7fec640ac970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.47:44544/0 (socket says 172.21.15.47:44544) 2024-03-29T11:23:24.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.915+0000 7fec63fff640 1 -- 172.21.15.47:0/2160607608 learned_addr learned my addr 172.21.15.47:0/2160607608 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:23:24.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.915+0000 7fec617fa640 10 client.?.objecter ms_handle_connect 0x7fec641422b0 2024-03-29T11:23:24.905 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.915+0000 7fec617fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:24.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.915+0000 7fec6b04d640 1 -- 172.21.15.47:0/2160607608 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec640b14d0 con 0x7fec64142690 2024-03-29T11:23:24.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.915+0000 7fec6b04d640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:24.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.915+0000 7fec6b04d640 10 monclient(hunting): authenticate will time out at 1448.103280s 2024-03-29T11:23:24.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.915+0000 7fec617fa640 10 client.?.objecter ms_handle_connect 0x7fec640b17d0 2024-03-29T11:23:24.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.915+0000 7fec617fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:24.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.915+0000 7fec617fa640 10 client.?.objecter ms_handle_connect 0x7fec64142690 2024-03-29T11:23:24.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.915+0000 7fec617fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:24.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.915+0000 7fec617fa640 1 -- 172.21.15.47:0/2160607608 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1445965133 0 0) 0x7fec640b11f0 con 0x7fec641422b0 2024-03-29T11:23:24.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.915+0000 7fec617fa640 10 monclient(hunting): _negotiate 2024-03-29T11:23:24.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.916+0000 7fec617fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:24.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.916+0000 7fec617fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:24.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.916+0000 7fec617fa640 10 monclient(hunting): my global_id is 4561 2024-03-29T11:23:24.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.916+0000 7fec617fa640 1 -- 172.21.15.47:0/2160607608 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec48004eb0 con 0x7fec641422b0 2024-03-29T11:23:24.906 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.916+0000 7fec617fa640 1 -- 172.21.15.47:0/2160607608 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3429784662 0 0) 0x7fec640b14d0 con 0x7fec64142690 2024-03-29T11:23:24.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.916+0000 7fec617fa640 10 monclient(hunting): _negotiate 2024-03-29T11:23:24.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.916+0000 7fec617fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:24.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.916+0000 7fec617fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:24.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.916+0000 7fec617fa640 10 monclient(hunting): my global_id is 4655 2024-03-29T11:23:24.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.916+0000 7fec617fa640 1 -- 172.21.15.47:0/2160607608 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec640b11f0 con 0x7fec64142690 2024-03-29T11:23:24.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.916+0000 7fec617fa640 1 -- 172.21.15.47:0/2160607608 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2839704388 0 0) 0x7fec640ac4e0 con 0x7fec640b17d0 2024-03-29T11:23:24.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.916+0000 7fec617fa640 10 monclient(hunting): _negotiate 2024-03-29T11:23:24.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.916+0000 7fec617fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:24.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.916+0000 7fec617fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:24.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.916+0000 7fec617fa640 10 monclient(hunting): my global_id is 4596 2024-03-29T11:23:24.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.916+0000 7fec617fa640 1 -- 172.21.15.47:0/2160607608 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec640b14d0 con 0x7fec640b17d0 2024-03-29T11:23:24.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.917+0000 7fec617fa640 1 -- 172.21.15.47:0/2160607608 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3257490296 0 0) 0x7fec48004eb0 con 0x7fec641422b0 2024-03-29T11:23:24.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.917+0000 7fec617fa640 1 -- 172.21.15.47:0/2160607608 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec640ac4e0 con 0x7fec641422b0 2024-03-29T11:23:24.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.917+0000 7fec617fa640 1 -- 172.21.15.47:0/2160607608 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fec58001930 con 0x7fec641422b0 2024-03-29T11:23:24.907 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.917+0000 7fec617fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:24.908 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.917+0000 7fec617fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:24.908 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.917+0000 7fec617fa640 10 monclient(hunting): dump: 2024-03-29T11:23:24.908 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:23:24.908 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:23:24.908 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:23:24.908 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:23:24.908 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:24.908 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:23:24.908 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:23:24.908 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:23:24.908 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:23:24.908 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:23:24.908 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.917+0000 7fec617fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:24.908 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.917+0000 7fec617fa640 1 -- 172.21.15.47:0/2160607608 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3573277287 0 0) 0x7fec640b11f0 con 0x7fec64142690 2024-03-29T11:23:24.909 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.917+0000 7fec617fa640 1 -- 172.21.15.47:0/2160607608 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec48004eb0 con 0x7fec64142690 2024-03-29T11:23:24.909 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.917+0000 7fec617fa640 1 -- 172.21.15.47:0/2160607608 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fec5c001830 con 0x7fec64142690 2024-03-29T11:23:24.909 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.917+0000 7fec617fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:24.909 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.917+0000 7fec617fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:24.909 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.917+0000 7fec617fa640 10 monclient(hunting): dump: 2024-03-29T11:23:24.909 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:23:24.909 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:23:24.909 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:23:24.909 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:23:24.909 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:24.909 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:23:24.909 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:23:24.909 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:23:24.909 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:23:24.910 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:23:24.910 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.917+0000 7fec617fa640 1 -- 172.21.15.47:0/2160607608 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2034170686 0 0) 0x7fec640b14d0 con 0x7fec640b17d0 2024-03-29T11:23:24.910 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.917+0000 7fec617fa640 1 -- 172.21.15.47:0/2160607608 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec640b11f0 con 0x7fec640b17d0 2024-03-29T11:23:24.910 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.917+0000 7fec617fa640 1 -- 172.21.15.47:0/2160607608 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fec50001830 con 0x7fec640b17d0 2024-03-29T11:23:24.910 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.917+0000 7fec617fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:24.910 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.917+0000 7fec617fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:24.910 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.917+0000 7fec617fa640 10 monclient(hunting): dump: 2024-03-29T11:23:24.910 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:23:24.910 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:23:24.910 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:23:24.910 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:23:24.910 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:24.910 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:23:24.910 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:23:24.911 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:23:24.911 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:23:24.911 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:23:24.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.918+0000 7fec617fa640 1 -- 172.21.15.47:0/2160607608 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3318558605 0 0) 0x7fec48004eb0 con 0x7fec64142690 2024-03-29T11:23:24.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.918+0000 7fec617fa640 1 -- 172.21.15.47:0/2160607608 >> v1:172.21.15.121:6789/0 conn(0x7fec641422b0 legacy=0x7fec640ac970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:24.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.918+0000 7fec617fa640 1 -- 172.21.15.47:0/2160607608 >> v1:172.21.15.47:6789/0 conn(0x7fec640b17d0 legacy=0x7fec640af750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:24.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.918+0000 7fec617fa640 10 monclient: _finish_hunting 0 2024-03-29T11:23:24.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.918+0000 7fec617fa640 1 monclient: found mon.c 2024-03-29T11:23:24.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.918+0000 7fec617fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:24.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.918+0000 7fec617fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:23:24.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.918+0000 7fec617fa640 1 -- 172.21.15.47:0/2160607608 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec640afe60 con 0x7fec64142690 2024-03-29T11:23:24.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.918+0000 7fec617fa640 10 monclient: _finish_auth 0 2024-03-29T11:23:24.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.918+0000 7fec617fa640 10 monclient: _check_auth_tickets 2024-03-29T11:23:24.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.918+0000 7fec617fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:24.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.918+0000 7fec6b04d640 5 monclient: authenticate success, global_id 4655 2024-03-29T11:23:24.911 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.918+0000 7fec6b04d640 10 monclient: _renew_subs 2024-03-29T11:23:24.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.918+0000 7fec6b04d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:23:24.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.918+0000 7fec6b04d640 1 -- 172.21.15.47:0/2160607608 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec640b00b0 con 0x7fec64142690 2024-03-29T11:23:24.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.918+0000 7fec6b04d640 10 client.4655.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:24.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.918+0000 7fec6b04d640 10 monclient: _renew_subs 2024-03-29T11:23:24.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.918+0000 7fec6b04d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:23:24.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.918+0000 7fec6b04d640 1 -- 172.21.15.47:0/2160607608 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fec641e2ee0 con 0x7fec64142690 2024-03-29T11:23:24.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.918+0000 7fec6b04d640 1 librados: init done 2024-03-29T11:23:24.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.918+0000 7fec617fa640 1 -- 172.21.15.47:0/2160607608 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec5c001a90 con 0x7fec64142690 2024-03-29T11:23:24.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.918+0000 7fec617fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:24.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.918+0000 7fec617fa640 1 -- 172.21.15.47:0/2160607608 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fec5c004e10 con 0x7fec64142690 2024-03-29T11:23:24.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.918+0000 7fec617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:24.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.918+0000 7fec617fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:24.912 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.918+0000 7fec617fa640 10 monclient: dump: 2024-03-29T11:23:24.912 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:23:24.912 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:23:24.913 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:23:24.913 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:23:24.913 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:24.913 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:23:24.913 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:23:24.913 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:23:24.913 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:23:24.913 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:23:24.913 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.918+0000 7fec627fc640 4 set_mon_vals no callback set 2024-03-29T11:23:24.913 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.919+0000 7fec6b04d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:24.913 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.919+0000 7fec6b04d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:24.913 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.919+0000 7fec6b04d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:23:24.913 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.919+0000 7fec6b04d640 1 -- 172.21.15.47:0/2160607608 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec44005080 con 0x7fec64142690 2024-03-29T11:23:24.913 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.921+0000 7fec617fa640 1 -- 172.21.15.47:0/2160607608 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7fec5c01ade0 con 0x7fec64142690 2024-03-29T11:23:24.913 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.921+0000 7fec617fa640 1 -- 172.21.15.47:0/2160607608 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3418925575 0 0) 0x7fec5c084d40 con 0x7fec64142690 2024-03-29T11:23:24.914 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.922+0000 7fec617fa640 10 client.4655.objecter ms_dispatch 0x7fec64000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:23:24.914 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.922+0000 7fec617fa640 3 client.4655.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:23:24.914 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.922+0000 7fec68de9640 10 monclient: get_auth_request con 0x7fec480769a0 auth_method 0 2024-03-29T11:23:24.914 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.922+0000 7fec617fa640 3 client.4655.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:23:24.914 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.922+0000 7fec617fa640 20 client.4655.objecter dump_active .. 0 homeless 2024-03-29T11:23:24.914 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.922+0000 7fec617fa640 10 client.4655.objecter ms_handle_connect 0x7fec480769a0 2024-03-29T11:23:24.914 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.929+0000 7fec617fa640 1 -- 172.21.15.47:0/2160607608 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fec5c050620 con 0x7fec64142690 2024-03-29T11:23:24.914 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.929+0000 7fec617fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:24.915 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:24.929+0000 7fec617fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:25.307 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:25.328+0000 7fec6b04d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:25.307 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:25.328+0000 7fec6b04d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:25.307 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:25.328+0000 7fec6b04d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:23:25.307 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:25.328+0000 7fec6b04d640 1 -- 172.21.15.47:0/2160607608 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fec44005330 con 0x7fec64142690 2024-03-29T11:23:25.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:25.329+0000 7fec617fa640 1 -- 172.21.15.47:0/2160607608 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1178522771) 0x7fec5c052550 con 0x7fec64142690 2024-03-29T11:23:25.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:25.329+0000 7fec617fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:25.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:25.329+0000 7fec617fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:25.308 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:23:25.309 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":11,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:22:45.925385+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:22:45.925385+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:25.312 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:25.333+0000 7fec6b04d640 10 librados: watch_flush enter 2024-03-29T11:23:25.312 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:25.333+0000 7fec6b04d640 10 librados: watch_flush exit 2024-03-29T11:23:25.312 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:25.333+0000 7fec6b04d640 20 client.4655.objecter shutdown clearing up homeless session... 2024-03-29T11:23:25.312 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:25.333+0000 7fec6b04d640 10 client.4655.objecter successfully canceled tick 2024-03-29T11:23:25.313 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:25.333+0000 7fec6b04d640 1 -- 172.21.15.47:0/2160607608 >> v1:172.21.15.121:6816/1964117370 conn(0x7fec480769a0 legacy=0x7fec48078e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:25.313 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:25.333+0000 7fec6b04d640 10 monclient: shutdown 2024-03-29T11:23:25.313 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:25.334+0000 7fec6b04d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:25.313 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:25.334+0000 7fec6b04d640 1 -- 172.21.15.47:0/2160607608 >> v1:172.21.15.47:6790/0 conn(0x7fec64142690 legacy=0x7fec640ad080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:25.313 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:25.334+0000 7fec6b04d640 1 -- 172.21.15.47:0/2160607608 shutdown_connections 2024-03-29T11:23:25.313 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:25.334+0000 7fec6b04d640 1 -- 172.21.15.47:0/2160607608 >> 172.21.15.47:0/2160607608 conn(0x7fec64066a00 msgr2=0x7fec6413ead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:25.313 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:25.334+0000 7fec6b04d640 1 -- 172.21.15.47:0/2160607608 shutdown_connections 2024-03-29T11:23:25.313 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:25.334+0000 7fec6b04d640 1 -- 172.21.15.47:0/2160607608 wait complete. 2024-03-29T11:23:25.313 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:25.334+0000 7fec6b04d640 1 librados: shutdown 2024-03-29T11:23:25.328 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:32.135 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:32.280 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.301+0000 7f889b59e640 10 monclient: get_monmap_and_config 2024-03-29T11:23:32.281 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.301+0000 7f889b59e640 10 monclient: build_initial_monmap 2024-03-29T11:23:32.281 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.301+0000 7f889b59e640 10 monclient: monmap: 2024-03-29T11:23:32.281 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:23:32.281 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:23:32.281 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:23:32.302495+0000 2024-03-29T11:23:32.281 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:23:32.302495+0000 2024-03-29T11:23:32.281 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:32.281 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:23:32.281 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:23:32.281 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:23:32.281 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:23:32.281 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:23:32.283 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.304+0000 7f889b59e640 1 Processor -- start 2024-03-29T11:23:32.283 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.304+0000 7f889b59e640 1 -- start start 2024-03-29T11:23:32.283 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.304+0000 7f889b59e640 10 monclient: init 2024-03-29T11:23:32.284 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.305+0000 7f889b59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:32.285 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.305+0000 7f889b59e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:23:32.288 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.309+0000 7f889b59e640 10 monclient(hunting): picked mon.noname-c con 0x7f8894006800 addr v1:172.21.15.121:6789/0 2024-03-29T11:23:32.290 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.309+0000 7f889b59e640 10 monclient(hunting): picked mon.noname-a con 0x7f8894006be0 addr v1:172.21.15.47:6789/0 2024-03-29T11:23:32.296 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.310+0000 7f889b59e640 10 monclient(hunting): picked mon.noname-b con 0x7f8894031b60 addr v1:172.21.15.47:6790/0 2024-03-29T11:23:32.296 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.310+0000 7f889b59e640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8894029370 con 0x7f8894006be0 2024-03-29T11:23:32.296 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.310+0000 7f889b59e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88940295e0 con 0x7f8894006800 2024-03-29T11:23:32.296 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.310+0000 7f889b59e640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8894031360 con 0x7f8894031b60 2024-03-29T11:23:32.296 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.310+0000 7f889b59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:32.296 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.310+0000 7f889b59e640 10 monclient(hunting): authenticate will time out at 1455.498150s 2024-03-29T11:23:32.296 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.310+0000 7f8899d9b640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f8894006be0 0x7f88940c4ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:45380/0 (socket says 172.21.15.47:45380) 2024-03-29T11:23:32.296 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.310+0000 7f8899d9b640 1 -- 172.21.15.47:0/2217213573 learned_addr learned my addr 172.21.15.47:0/2217213573 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:23:32.296 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.311+0000 7f889959a640 1 -- 172.21.15.47:0/2217213573 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3516586587 0 0) 0x7f8894029370 con 0x7f8894006be0 2024-03-29T11:23:32.296 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.311+0000 7f889959a640 10 monclient(hunting): _negotiate 2024-03-29T11:23:32.296 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.311+0000 7f889959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:32.297 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.311+0000 7f889959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:32.297 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.311+0000 7f889959a640 10 monclient(hunting): my global_id is 4599 2024-03-29T11:23:32.297 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.311+0000 7f889959a640 1 -- 172.21.15.47:0/2217213573 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8880003520 con 0x7f8894006be0 2024-03-29T11:23:32.297 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.311+0000 7f889959a640 1 -- 172.21.15.47:0/2217213573 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 695861135 0 0) 0x7f8880003520 con 0x7f8894006be0 2024-03-29T11:23:32.297 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.311+0000 7f889959a640 1 -- 172.21.15.47:0/2217213573 >> v1:172.21.15.47:6790/0 conn(0x7f8894031b60 legacy=0x7f88940d30c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:23:32.297 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.315+0000 7f889959a640 1 -- 172.21.15.47:0/2217213573 >> v1:172.21.15.121:6789/0 conn(0x7f8894006800 legacy=0x7f88940cad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:32.297 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.315+0000 7f889959a640 10 monclient: _finish_hunting 0 2024-03-29T11:23:32.297 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.315+0000 7f889959a640 1 monclient: found mon.noname-a 2024-03-29T11:23:32.297 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.315+0000 7f889959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:32.297 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.315+0000 7f889959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-03-29T11:23:32.297 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.315+0000 7f889959a640 1 -- 172.21.15.47:0/2217213573 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8894031640 con 0x7f8894006be0 2024-03-29T11:23:32.297 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.315+0000 7f889959a640 10 monclient: _finish_auth 0 2024-03-29T11:23:32.297 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.315+0000 7f889959a640 10 monclient: _check_auth_tickets 2024-03-29T11:23:32.297 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.315+0000 7f889959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:32.297 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.316+0000 7f889959a640 1 -- 172.21.15.47:0/2217213573 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f8888001670 con 0x7f8894006be0 2024-03-29T11:23:32.298 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.316+0000 7f889959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:32.298 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.316+0000 7f889959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:23:32.298 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.316+0000 7f889959a640 10 monclient: dump: 2024-03-29T11:23:32.298 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:23:32.298 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:23:32.298 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:23:32.298 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:23:32.298 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:32.298 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:23:32.298 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:23:32.298 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:23:32.298 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:23:32.298 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:23:32.299 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.316+0000 7f889b59e640 5 monclient: authenticate success, global_id 4599 2024-03-29T11:23:32.299 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.316+0000 7f889b59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:32.300 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.318+0000 7f889959a640 1 -- 172.21.15.47:0/2217213573 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f88880018d0 con 0x7f8894006be0 2024-03-29T11:23:32.300 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.318+0000 7f889959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:32.300 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.318+0000 7f889b59e640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:32.300 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.318+0000 7f889b59e640 4 set_mon_vals no callback set 2024-03-29T11:23:32.300 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.318+0000 7f889959a640 1 -- 172.21.15.47:0/2217213573 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f8888001ca0 con 0x7f8894006be0 2024-03-29T11:23:32.300 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.318+0000 7f889959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:32.300 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.318+0000 7f889959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:32.300 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.318+0000 7f889959a640 10 monclient: dump: 2024-03-29T11:23:32.300 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:23:32.300 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:23:32.300 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:23:32.300 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:23:32.300 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:32.301 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:23:32.301 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:23:32.301 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:23:32.301 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:23:32.301 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:23:32.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.319+0000 7f889b59e640 10 monclient: shutdown 2024-03-29T11:23:32.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.319+0000 7f889b59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:32.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.319+0000 7f889b59e640 1 -- 172.21.15.47:0/2217213573 >> v1:172.21.15.47:6789/0 conn(0x7f8894006be0 legacy=0x7f88940c4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:32.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.319+0000 7f889b59e640 1 -- 172.21.15.47:0/2217213573 shutdown_connections 2024-03-29T11:23:32.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.319+0000 7f889b59e640 1 -- 172.21.15.47:0/2217213573 >> 172.21.15.47:0/2217213573 conn(0x7f88940058f0 msgr2=0x7f8894005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:32.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.319+0000 7f889b59e640 1 -- 172.21.15.47:0/2217213573 shutdown_connections 2024-03-29T11:23:32.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.319+0000 7f889b59e640 1 -- 172.21.15.47:0/2217213573 wait complete. 2024-03-29T11:23:32.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.320+0000 7f889b59e640 10 monclient: build_initial_monmap 2024-03-29T11:23:32.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.320+0000 7f889b59e640 10 monclient: monmap: 2024-03-29T11:23:32.301 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:23:32.302 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:32.302 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:23:32.302 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:23:32.302 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:32.302 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:23:32.302 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:23:32.302 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:23:32.302 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:23:32.302 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:23:32.303 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.321+0000 7f889b59e640 1 librados: starting msgr at 2024-03-29T11:23:32.303 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.321+0000 7f889b59e640 1 librados: starting objecter 2024-03-29T11:23:32.303 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.321+0000 7f889b59e640 1 Processor -- start 2024-03-29T11:23:32.303 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.321+0000 7f889b59e640 1 -- start start 2024-03-29T11:23:32.303 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.321+0000 7f889b59e640 1 librados: setting wanted keys 2024-03-29T11:23:32.303 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.321+0000 7f889b59e640 1 librados: calling monclient init 2024-03-29T11:23:32.303 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.321+0000 7f889b59e640 10 monclient: init 2024-03-29T11:23:32.303 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.322+0000 7f889b59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:32.303 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.322+0000 7f889b59e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:23:32.303 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.323+0000 7f889b59e640 10 monclient(hunting): picked mon.noname-a con 0x7f8894006800 addr v1:172.21.15.47:6789/0 2024-03-29T11:23:32.304 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.323+0000 7f889b59e640 10 monclient(hunting): picked mon.noname-c con 0x7f8894031b60 addr v1:172.21.15.47:6790/0 2024-03-29T11:23:32.304 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.324+0000 7f889a59c640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f8894006800 0x7f889415b9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:36626/0 (socket says 172.21.15.47:36626) 2024-03-29T11:23:32.309 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.325+0000 7f889b59e640 10 monclient(hunting): picked mon.noname-b con 0x7f88940bd750 addr v1:172.21.15.121:6789/0 2024-03-29T11:23:32.309 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.325+0000 7f889b59e640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f889415c570 con 0x7f8894006800 2024-03-29T11:23:32.309 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.325+0000 7f889b59e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88940c55a0 con 0x7f88940bd750 2024-03-29T11:23:32.309 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.325+0000 7f889b59e640 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8894155d40 con 0x7f8894031b60 2024-03-29T11:23:32.309 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.325+0000 7f889b59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:32.309 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.325+0000 7f889b59e640 10 monclient(hunting): authenticate will time out at 1455.513322s 2024-03-29T11:23:32.309 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.326+0000 7f889a59c640 1 -- 172.21.15.47:0/1911705590 learned_addr learned my addr 172.21.15.47:0/1911705590 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:23:32.310 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.328+0000 7f8892ffd640 10 client.?.objecter ms_handle_connect 0x7f8894006800 2024-03-29T11:23:32.310 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.328+0000 7f8892ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:32.310 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.328+0000 7f8892ffd640 10 client.?.objecter ms_handle_connect 0x7f8894031b60 2024-03-29T11:23:32.310 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.328+0000 7f8892ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:32.310 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.328+0000 7f8892ffd640 1 -- 172.21.15.47:0/1911705590 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 658208315 0 0) 0x7f889415c570 con 0x7f8894006800 2024-03-29T11:23:32.310 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.328+0000 7f8892ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:32.310 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.328+0000 7f8892ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:32.310 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.328+0000 7f8892ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:32.310 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.328+0000 7f8892ffd640 10 monclient(hunting): my global_id is 4602 2024-03-29T11:23:32.310 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.328+0000 7f8892ffd640 1 -- 172.21.15.47:0/1911705590 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8884004eb0 con 0x7f8894006800 2024-03-29T11:23:32.310 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.328+0000 7f8892ffd640 1 -- 172.21.15.47:0/1911705590 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3533136924 0 0) 0x7f8894155d40 con 0x7f8894031b60 2024-03-29T11:23:32.310 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.328+0000 7f8892ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:32.310 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.328+0000 7f8892ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:32.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.328+0000 7f8892ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:32.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.328+0000 7f8892ffd640 10 monclient(hunting): my global_id is 4658 2024-03-29T11:23:32.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.328+0000 7f8892ffd640 1 -- 172.21.15.47:0/1911705590 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f889415c570 con 0x7f8894031b60 2024-03-29T11:23:32.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.328+0000 7f8892ffd640 1 -- 172.21.15.47:0/1911705590 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4226722486 0 0) 0x7f8884004eb0 con 0x7f8894006800 2024-03-29T11:23:32.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.328+0000 7f8892ffd640 1 -- 172.21.15.47:0/1911705590 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8894155d40 con 0x7f8894006800 2024-03-29T11:23:32.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.328+0000 7f8892ffd640 1 -- 172.21.15.47:0/1911705590 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f888c001930 con 0x7f8894006800 2024-03-29T11:23:32.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.328+0000 7f8892ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:32.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.328+0000 7f8892ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:23:32.311 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.328+0000 7f8892ffd640 10 monclient(hunting): dump: 2024-03-29T11:23:32.311 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:23:32.311 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:23:32.311 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:23:32.311 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:23:32.311 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:32.311 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:23:32.312 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:23:32.312 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:23:32.312 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:23:32.312 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:23:32.312 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.328+0000 7f8892ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:32.312 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.329+0000 7f8892ffd640 10 client.?.objecter ms_handle_connect 0x7f88940bd750 2024-03-29T11:23:32.312 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.329+0000 7f8892ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:32.312 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.329+0000 7f8892ffd640 1 -- 172.21.15.47:0/1911705590 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2875247147 0 0) 0x7f889415c570 con 0x7f8894031b60 2024-03-29T11:23:32.312 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.329+0000 7f8892ffd640 1 -- 172.21.15.47:0/1911705590 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8884004eb0 con 0x7f8894031b60 2024-03-29T11:23:32.312 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.329+0000 7f8892ffd640 1 -- 172.21.15.47:0/1911705590 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f8888001be0 con 0x7f8894031b60 2024-03-29T11:23:32.312 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.329+0000 7f8892ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:32.312 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.329+0000 7f8892ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:32.312 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.329+0000 7f8892ffd640 10 monclient(hunting): dump: 2024-03-29T11:23:32.312 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:23:32.312 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:23:32.313 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:23:32.313 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:23:32.313 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:32.313 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:23:32.313 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:23:32.313 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:23:32.313 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:23:32.313 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:23:32.313 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.329+0000 7f8892ffd640 1 -- 172.21.15.47:0/1911705590 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1935033577 0 0) 0x7f88940c55a0 con 0x7f88940bd750 2024-03-29T11:23:32.313 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.329+0000 7f8892ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:32.313 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.329+0000 7f8892ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:32.313 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.329+0000 7f8892ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:32.313 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.329+0000 7f8892ffd640 10 monclient(hunting): my global_id is 4567 2024-03-29T11:23:32.313 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.329+0000 7f8892ffd640 1 -- 172.21.15.47:0/1911705590 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f889415c570 con 0x7f88940bd750 2024-03-29T11:23:32.316 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.329+0000 7f8892ffd640 1 -- 172.21.15.47:0/1911705590 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2585220707 0 0) 0x7f8894155d40 con 0x7f8894006800 2024-03-29T11:23:32.316 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.329+0000 7f8892ffd640 1 -- 172.21.15.47:0/1911705590 >> v1:172.21.15.47:6790/0 conn(0x7f8894031b60 legacy=0x7f88940bd040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:32.316 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.329+0000 7f8892ffd640 1 -- 172.21.15.47:0/1911705590 >> v1:172.21.15.121:6789/0 conn(0x7f88940bd750 legacy=0x7f88940c6230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:32.316 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.329+0000 7f8892ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:23:32.316 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.329+0000 7f8892ffd640 1 monclient: found mon.a 2024-03-29T11:23:32.316 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.329+0000 7f8892ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:32.316 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.329+0000 7f8892ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:23:32.316 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.329+0000 7f8892ffd640 1 -- 172.21.15.47:0/1911705590 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f889415c100 con 0x7f8894006800 2024-03-29T11:23:32.316 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.329+0000 7f8892ffd640 10 monclient: _finish_auth 0 2024-03-29T11:23:32.316 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.329+0000 7f8892ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:23:32.316 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.329+0000 7f8892ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:32.317 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.331+0000 7f8892ffd640 1 -- 172.21.15.47:0/1911705590 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f888c001b10 con 0x7f8894006800 2024-03-29T11:23:32.317 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.331+0000 7f8892ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:32.317 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.331+0000 7f8892ffd640 1 -- 172.21.15.47:0/1911705590 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7f888c0038c0 con 0x7f8894006800 2024-03-29T11:23:32.317 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.331+0000 7f8892ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:32.317 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.331+0000 7f8892ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:32.317 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.331+0000 7f8892ffd640 10 monclient: dump: 2024-03-29T11:23:32.317 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:23:32.317 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:23:32.317 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:23:32.317 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:23:32.317 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:32.317 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:23:32.317 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:23:32.317 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:23:32.317 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:23:32.318 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:23:32.318 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.331+0000 7f8898d99640 4 set_mon_vals no callback set 2024-03-29T11:23:32.318 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.331+0000 7f889b59e640 5 monclient: authenticate success, global_id 4602 2024-03-29T11:23:32.318 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.331+0000 7f889b59e640 10 monclient: _renew_subs 2024-03-29T11:23:32.318 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.331+0000 7f889b59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:23:32.318 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.331+0000 7f889b59e640 1 -- 172.21.15.47:0/1911705590 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f889415c2f0 con 0x7f8894006800 2024-03-29T11:23:32.318 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.332+0000 7f889b59e640 10 client.4602.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:32.318 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.332+0000 7f889b59e640 10 monclient: _renew_subs 2024-03-29T11:23:32.318 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.332+0000 7f889b59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:23:32.318 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.332+0000 7f889b59e640 1 -- 172.21.15.47:0/1911705590 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f88940c6b20 con 0x7f8894006800 2024-03-29T11:23:32.318 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.332+0000 7f889b59e640 1 librados: init done 2024-03-29T11:23:32.318 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.333+0000 7f8890ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:32.318 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.333+0000 7f8890ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:32.318 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.333+0000 7f8890ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:23:32.319 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.333+0000 7f8890ff9640 1 -- 172.21.15.47:0/1911705590 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f887c005080 con 0x7f8894006800 2024-03-29T11:23:32.319 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.334+0000 7f8892ffd640 1 -- 172.21.15.47:0/1911705590 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7f888c003aa0 con 0x7f8894006800 2024-03-29T11:23:32.319 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.335+0000 7f8892ffd640 1 -- 172.21.15.47:0/1911705590 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3418925575 0 0) 0x7f888c084b50 con 0x7f8894006800 2024-03-29T11:23:32.319 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.335+0000 7f8892ffd640 10 client.4602.objecter ms_dispatch 0x7f889c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:23:32.319 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.335+0000 7f8892ffd640 3 client.4602.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:23:32.319 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.335+0000 7f8892ffd640 3 client.4602.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:23:32.319 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.335+0000 7f8892ffd640 20 client.4602.objecter dump_active .. 0 homeless 2024-03-29T11:23:32.319 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.335+0000 7f8899d9b640 10 monclient: get_auth_request con 0x7f888406dd60 auth_method 0 2024-03-29T11:23:32.319 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.336+0000 7f8892ffd640 10 client.4602.objecter ms_handle_connect 0x7f888406dd60 2024-03-29T11:23:32.325 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.346+0000 7f8892ffd640 1 -- 172.21.15.47:0/1911705590 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f888c050260 con 0x7f8894006800 2024-03-29T11:23:32.325 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.346+0000 7f8892ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:32.325 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.346+0000 7f8892ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:32.667 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.686+0000 7f8890ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:32.667 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.686+0000 7f8890ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:32.667 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.686+0000 7f8890ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-03-29T11:23:32.667 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.686+0000 7f8890ff9640 1 -- 172.21.15.47:0/1911705590 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f887c005330 con 0x7f8894006800 2024-03-29T11:23:32.668 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.689+0000 7f8892ffd640 1 -- 172.21.15.47:0/1911705590 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1178522771) 0x7f888c052360 con 0x7f8894006800 2024-03-29T11:23:32.668 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.689+0000 7f8892ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:32.668 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.689+0000 7f8892ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:32.671 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:23:32.672 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":11,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:22:45.925385+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:22:45.925385+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:32.675 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.696+0000 7f8890ff9640 10 librados: watch_flush enter 2024-03-29T11:23:32.675 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.696+0000 7f8890ff9640 10 librados: watch_flush exit 2024-03-29T11:23:32.675 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.696+0000 7f8890ff9640 20 client.4602.objecter shutdown clearing up homeless session... 2024-03-29T11:23:32.675 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.696+0000 7f8890ff9640 10 client.4602.objecter successfully canceled tick 2024-03-29T11:23:32.676 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.696+0000 7f8890ff9640 1 -- 172.21.15.47:0/1911705590 >> v1:172.21.15.121:6816/1964117370 conn(0x7f888406dd60 legacy=0x7f8884070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:32.676 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.696+0000 7f8890ff9640 10 monclient: shutdown 2024-03-29T11:23:32.676 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.696+0000 7f8890ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:32.676 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.696+0000 7f8890ff9640 1 -- 172.21.15.47:0/1911705590 >> v1:172.21.15.47:6789/0 conn(0x7f8894006800 legacy=0x7f889415b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:32.676 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.696+0000 7f8890ff9640 1 -- 172.21.15.47:0/1911705590 shutdown_connections 2024-03-29T11:23:32.676 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.696+0000 7f8890ff9640 1 -- 172.21.15.47:0/1911705590 >> 172.21.15.47:0/1911705590 conn(0x7f88940058f0 msgr2=0x7f88940c4990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:32.682 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.700+0000 7f8890ff9640 1 -- 172.21.15.47:0/1911705590 shutdown_connections 2024-03-29T11:23:32.682 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.702+0000 7f8890ff9640 1 -- 172.21.15.47:0/1911705590 wait complete. 2024-03-29T11:23:32.682 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:32.703+0000 7f8890ff9640 1 librados: shutdown 2024-03-29T11:23:32.702 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:39.509 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:39.648 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.668+0000 7fb83bc55640 10 monclient: get_monmap_and_config 2024-03-29T11:23:39.648 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.668+0000 7fb83bc55640 10 monclient: build_initial_monmap 2024-03-29T11:23:39.648 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.668+0000 7fb83bc55640 10 monclient: monmap: 2024-03-29T11:23:39.648 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:23:39.648 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:23:39.648 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:23:39.669510+0000 2024-03-29T11:23:39.648 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:23:39.669510+0000 2024-03-29T11:23:39.648 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:39.649 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:23:39.649 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:23:39.649 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-c 2024-03-29T11:23:39.649 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-03-29T11:23:39.649 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:23:39.650 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.669+0000 7fb83bc55640 1 Processor -- start 2024-03-29T11:23:39.651 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.669+0000 7fb83bc55640 1 -- start start 2024-03-29T11:23:39.651 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.669+0000 7fb83bc55640 10 monclient: init 2024-03-29T11:23:39.651 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.670+0000 7fb83bc55640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:39.651 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.671+0000 7fb83bc55640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:23:39.651 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.671+0000 7fb83bc55640 10 monclient(hunting): picked mon.noname-b con 0x7fb8341457f0 addr v1:172.21.15.47:6790/0 2024-03-29T11:23:39.651 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.671+0000 7fb83bc55640 10 monclient(hunting): picked mon.noname-a con 0x7fb83413f650 addr v1:172.21.15.47:6789/0 2024-03-29T11:23:39.651 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.671+0000 7fb8399f1640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fb8341457f0 0x7fb83413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:49472/0 (socket says 172.21.15.47:49472) 2024-03-29T11:23:39.651 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.671+0000 7fb8399f1640 1 -- 172.21.15.47:0/3642275023 learned_addr learned my addr 172.21.15.47:0/3642275023 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:23:39.651 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.671+0000 7fb83bc55640 10 monclient(hunting): picked mon.noname-c con 0x7fb8340a9d70 addr v1:172.21.15.121:6789/0 2024-03-29T11:23:39.651 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.671+0000 7fb83bc55640 1 -- 172.21.15.47:0/3642275023 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8340aaef0 con 0x7fb83413f650 2024-03-29T11:23:39.651 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.671+0000 7fb83bc55640 1 -- 172.21.15.47:0/3642275023 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb834145bd0 con 0x7fb8340a9d70 2024-03-29T11:23:39.654 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.671+0000 7fb83bc55640 1 -- 172.21.15.47:0/3642275023 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb834145eb0 con 0x7fb8341457f0 2024-03-29T11:23:39.654 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.672+0000 7fb83bc55640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:39.654 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.672+0000 7fb83bc55640 10 monclient(hunting): authenticate will time out at 1462.860182s 2024-03-29T11:23:39.654 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.672+0000 7fb8389ef640 1 -- 172.21.15.47:0/3642275023 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1946203293 0 0) 0x7fb834145bd0 con 0x7fb8340a9d70 2024-03-29T11:23:39.654 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.672+0000 7fb8389ef640 10 monclient(hunting): _negotiate 2024-03-29T11:23:39.654 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.672+0000 7fb8389ef640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:39.655 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.672+0000 7fb8389ef640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:39.655 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.672+0000 7fb8389ef640 10 monclient(hunting): my global_id is 4570 2024-03-29T11:23:39.655 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.672+0000 7fb8389ef640 1 -- 172.21.15.47:0/3642275023 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb820003520 con 0x7fb8340a9d70 2024-03-29T11:23:39.655 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.672+0000 7fb8389ef640 1 -- 172.21.15.47:0/3642275023 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3586030996 0 0) 0x7fb834145eb0 con 0x7fb8341457f0 2024-03-29T11:23:39.655 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.672+0000 7fb8389ef640 10 monclient(hunting): _negotiate 2024-03-29T11:23:39.655 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.672+0000 7fb8389ef640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:39.655 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.672+0000 7fb8389ef640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:39.655 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.672+0000 7fb8389ef640 10 monclient(hunting): my global_id is 4661 2024-03-29T11:23:39.655 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.672+0000 7fb8389ef640 1 -- 172.21.15.47:0/3642275023 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb834145bd0 con 0x7fb8341457f0 2024-03-29T11:23:39.655 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.672+0000 7fb8389ef640 1 -- 172.21.15.47:0/3642275023 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 908860030 0 0) 0x7fb834145bd0 con 0x7fb8341457f0 2024-03-29T11:23:39.655 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.673+0000 7fb8389ef640 1 -- 172.21.15.47:0/3642275023 >> v1:172.21.15.121:6789/0 conn(0x7fb8340a9d70 legacy=0x7fb83414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:39.655 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.673+0000 7fb8389ef640 1 -- 172.21.15.47:0/3642275023 >> v1:172.21.15.47:6789/0 conn(0x7fb83413f650 legacy=0x7fb8340b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:23:39.655 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.673+0000 7fb8389ef640 10 monclient: _finish_hunting 0 2024-03-29T11:23:39.655 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.673+0000 7fb8389ef640 1 monclient: found mon.noname-b 2024-03-29T11:23:39.656 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.673+0000 7fb8389ef640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:39.656 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.673+0000 7fb8389ef640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-03-29T11:23:39.656 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.673+0000 7fb8389ef640 1 -- 172.21.15.47:0/3642275023 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb834146340 con 0x7fb8341457f0 2024-03-29T11:23:39.656 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.673+0000 7fb8389ef640 10 monclient: _finish_auth 0 2024-03-29T11:23:39.656 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.673+0000 7fb8389ef640 10 monclient: _check_auth_tickets 2024-03-29T11:23:39.656 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.673+0000 7fb8389ef640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:39.656 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.673+0000 7fb8389ef640 1 -- 172.21.15.47:0/3642275023 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fb8240015f0 con 0x7fb8341457f0 2024-03-29T11:23:39.656 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.673+0000 7fb8389ef640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:39.656 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.673+0000 7fb8389ef640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:39.656 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.673+0000 7fb8389ef640 10 monclient: dump: 2024-03-29T11:23:39.656 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:23:39.656 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:23:39.656 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:23:39.657 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:23:39.657 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:39.657 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:23:39.657 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:23:39.657 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:23:39.657 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:23:39.657 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:23:39.657 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.674+0000 7fb83bc55640 5 monclient: authenticate success, global_id 4661 2024-03-29T11:23:39.657 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.674+0000 7fb8389ef640 1 -- 172.21.15.47:0/3642275023 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb824001750 con 0x7fb8341457f0 2024-03-29T11:23:39.657 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.674+0000 7fb83bc55640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:39.657 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.674+0000 7fb8389ef640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:39.657 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.674+0000 7fb8389ef640 1 -- 172.21.15.47:0/3642275023 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fb824001ba0 con 0x7fb8341457f0 2024-03-29T11:23:39.657 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.674+0000 7fb83bc55640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:39.657 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.674+0000 7fb83bc55640 4 set_mon_vals no callback set 2024-03-29T11:23:39.661 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.674+0000 7fb8389ef640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:39.661 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.675+0000 7fb8389ef640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:39.661 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.675+0000 7fb8389ef640 10 monclient: dump: 2024-03-29T11:23:39.661 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:23:39.661 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:23:39.661 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:23:39.662 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:23:39.662 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:39.662 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:23:39.662 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:23:39.662 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:23:39.662 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:23:39.662 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:23:39.662 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.676+0000 7fb83bc55640 10 monclient: shutdown 2024-03-29T11:23:39.662 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.676+0000 7fb83bc55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:39.662 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.676+0000 7fb83bc55640 1 -- 172.21.15.47:0/3642275023 >> v1:172.21.15.47:6790/0 conn(0x7fb8341457f0 legacy=0x7fb83413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:39.662 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.676+0000 7fb83bc55640 1 -- 172.21.15.47:0/3642275023 shutdown_connections 2024-03-29T11:23:39.662 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.676+0000 7fb83bc55640 1 -- 172.21.15.47:0/3642275023 >> 172.21.15.47:0/3642275023 conn(0x7fb834066a00 msgr2=0x7fb834066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:39.662 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.677+0000 7fb83bc55640 1 -- 172.21.15.47:0/3642275023 shutdown_connections 2024-03-29T11:23:39.662 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.677+0000 7fb83bc55640 1 -- 172.21.15.47:0/3642275023 wait complete. 2024-03-29T11:23:39.663 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.678+0000 7fb83bc55640 10 monclient: build_initial_monmap 2024-03-29T11:23:39.663 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.678+0000 7fb83bc55640 10 monclient: monmap: 2024-03-29T11:23:39.663 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-29T11:23:39.663 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:39.663 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-29T11:23:39.663 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-29T11:23:39.663 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:39.663 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:23:39.663 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-03-29T11:23:39.663 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.noname-b 2024-03-29T11:23:39.663 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-03-29T11:23:39.663 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:23:39.663 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.678+0000 7fb83bc55640 1 librados: starting msgr at 2024-03-29T11:23:39.664 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.678+0000 7fb83bc55640 1 librados: starting objecter 2024-03-29T11:23:39.664 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.678+0000 7fb83bc55640 1 Processor -- start 2024-03-29T11:23:39.664 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.679+0000 7fb83bc55640 1 -- start start 2024-03-29T11:23:39.664 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.679+0000 7fb83bc55640 1 librados: setting wanted keys 2024-03-29T11:23:39.664 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.679+0000 7fb83bc55640 1 librados: calling monclient init 2024-03-29T11:23:39.664 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.679+0000 7fb83bc55640 10 monclient: init 2024-03-29T11:23:39.664 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.680+0000 7fb83bc55640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:39.664 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.680+0000 7fb83bc55640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:23:39.664 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.680+0000 7fb83bc55640 10 monclient(hunting): picked mon.noname-b con 0x7fb8340a9d70 addr v1:172.21.15.121:6789/0 2024-03-29T11:23:39.664 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.680+0000 7fb83bc55640 10 monclient(hunting): picked mon.noname-a con 0x7fb83413f650 addr v1:172.21.15.47:6789/0 2024-03-29T11:23:39.664 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.680+0000 7fb8399f1640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fb8340a9d70 0x7fb834136e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.47:39822/0 (socket says 172.21.15.47:39822) 2024-03-29T11:23:39.664 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.680+0000 7fb83a1f2640 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fb8341457f0 0x7fb8341dab50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:34946/0 (socket says 172.21.15.47:34946) 2024-03-29T11:23:39.664 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.680+0000 7fb8399f1640 1 -- 172.21.15.47:0/2952316097 learned_addr learned my addr 172.21.15.47:0/2952316097 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-29T11:23:39.665 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.680+0000 7fb83bc55640 10 monclient(hunting): picked mon.noname-c con 0x7fb8341457f0 addr v1:172.21.15.47:6790/0 2024-03-29T11:23:39.665 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.680+0000 7fb83bc55640 1 -- 172.21.15.47:0/2952316097 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8341428a0 con 0x7fb83413f650 2024-03-29T11:23:39.665 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.680+0000 7fb82a7fc640 10 client.?.objecter ms_handle_connect 0x7fb8341457f0 2024-03-29T11:23:39.665 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.680+0000 7fb82a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:39.665 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.680+0000 7fb82a7fc640 10 client.?.objecter ms_handle_connect 0x7fb8340a9d70 2024-03-29T11:23:39.665 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.680+0000 7fb82a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:39.665 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.680+0000 7fb83bc55640 1 -- 172.21.15.47:0/2952316097 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb834141ed0 con 0x7fb8340a9d70 2024-03-29T11:23:39.665 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.680+0000 7fb83bc55640 1 -- 172.21.15.47:0/2952316097 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb834141a30 con 0x7fb8341457f0 2024-03-29T11:23:39.665 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.680+0000 7fb83bc55640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:39.665 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.680+0000 7fb83bc55640 10 monclient(hunting): authenticate will time out at 1462.868775s 2024-03-29T11:23:39.665 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.681+0000 7fb82a7fc640 1 -- 172.21.15.47:0/2952316097 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3382477350 0 0) 0x7fb834141ed0 con 0x7fb8340a9d70 2024-03-29T11:23:39.665 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.681+0000 7fb82a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:23:39.665 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.681+0000 7fb82a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:39.665 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.681+0000 7fb82a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:39.665 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.681+0000 7fb82a7fc640 10 monclient(hunting): my global_id is 4573 2024-03-29T11:23:39.666 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.681+0000 7fb82a7fc640 1 -- 172.21.15.47:0/2952316097 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb818004eb0 con 0x7fb8340a9d70 2024-03-29T11:23:39.666 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.681+0000 7fb82a7fc640 1 -- 172.21.15.47:0/2952316097 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3891714626 0 0) 0x7fb834141a30 con 0x7fb8341457f0 2024-03-29T11:23:39.666 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.681+0000 7fb82a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:23:39.666 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.681+0000 7fb82a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:39.666 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.681+0000 7fb82a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:39.666 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.681+0000 7fb82a7fc640 10 monclient(hunting): my global_id is 4664 2024-03-29T11:23:39.666 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.681+0000 7fb82a7fc640 1 -- 172.21.15.47:0/2952316097 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb834141ed0 con 0x7fb8341457f0 2024-03-29T11:23:39.668 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.681+0000 7fb82a7fc640 1 -- 172.21.15.47:0/2952316097 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3857059436 0 0) 0x7fb834141ed0 con 0x7fb8341457f0 2024-03-29T11:23:39.668 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.681+0000 7fb82a7fc640 1 -- 172.21.15.47:0/2952316097 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb834141a30 con 0x7fb8341457f0 2024-03-29T11:23:39.668 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.681+0000 7fb82a7fc640 1 -- 172.21.15.47:0/2952316097 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fb82c001bc0 con 0x7fb8341457f0 2024-03-29T11:23:39.668 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.681+0000 7fb82a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:39.668 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.681+0000 7fb82a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:39.668 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.681+0000 7fb82a7fc640 10 monclient(hunting): dump: 2024-03-29T11:23:39.668 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:23:39.668 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:23:39.668 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:23:39.669 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:23:39.669 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:39.669 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:23:39.669 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:23:39.669 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:23:39.669 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:23:39.669 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:23:39.669 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.681+0000 7fb82a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:39.669 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.681+0000 7fb82a7fc640 1 -- 172.21.15.47:0/2952316097 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3759648160 0 0) 0x7fb834141a30 con 0x7fb8341457f0 2024-03-29T11:23:39.669 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.681+0000 7fb82a7fc640 1 -- 172.21.15.47:0/2952316097 >> v1:172.21.15.121:6789/0 conn(0x7fb8340a9d70 legacy=0x7fb834136e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:39.669 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.681+0000 7fb82a7fc640 1 -- 172.21.15.47:0/2952316097 >> v1:172.21.15.47:6789/0 conn(0x7fb83413f650 legacy=0x7fb8341da440 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:23:39.669 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.684+0000 7fb82a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:23:39.669 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.684+0000 7fb82a7fc640 1 monclient: found mon.c 2024-03-29T11:23:39.669 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.684+0000 7fb82a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:39.669 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.684+0000 7fb82a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:23:39.670 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.684+0000 7fb82a7fc640 1 -- 172.21.15.47:0/2952316097 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb834141bc0 con 0x7fb8341457f0 2024-03-29T11:23:39.670 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.684+0000 7fb82a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:23:39.670 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.684+0000 7fb82a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:23:39.670 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.684+0000 7fb82a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:39.670 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.684+0000 7fb83bc55640 5 monclient: authenticate success, global_id 4664 2024-03-29T11:23:39.670 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.684+0000 7fb83bc55640 10 monclient: _renew_subs 2024-03-29T11:23:39.670 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.684+0000 7fb83bc55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:23:39.670 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.684+0000 7fb83bc55640 1 -- 172.21.15.47:0/2952316097 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb834137540 con 0x7fb8341457f0 2024-03-29T11:23:39.670 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.684+0000 7fb83bc55640 10 client.4664.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:39.670 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.684+0000 7fb83bc55640 10 monclient: _renew_subs 2024-03-29T11:23:39.670 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.684+0000 7fb83bc55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:23:39.670 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.684+0000 7fb83bc55640 1 -- 172.21.15.47:0/2952316097 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8341447c0 con 0x7fb8341457f0 2024-03-29T11:23:39.670 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.685+0000 7fb83bc55640 1 librados: init done 2024-03-29T11:23:39.670 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.685+0000 7fb82a7fc640 1 -- 172.21.15.47:0/2952316097 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb82c0020b0 con 0x7fb8341457f0 2024-03-29T11:23:39.670 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.685+0000 7fb82a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:39.671 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.685+0000 7fb82a7fc640 1 -- 172.21.15.47:0/2952316097 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 616385611 0 0) 0x7fb82c003b30 con 0x7fb8341457f0 2024-03-29T11:23:39.671 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.685+0000 7fb82a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:39.671 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.685+0000 7fb82a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:39.671 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.685+0000 7fb82a7fc640 10 monclient: dump: 2024-03-29T11:23:39.671 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-29T11:23:39.671 INFO:teuthology.misc.health.smithi047.stderr:fsid bb938893-5bcf-496d-8a49-24fbf9065510 2024-03-29T11:23:39.671 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-29T11:12:04.734825+0000 2024-03-29T11:23:39.671 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-29T11:12:04.734825+0000 2024-03-29T11:23:39.671 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:39.671 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-29T11:23:39.671 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-03-29T11:23:39.671 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.121:6789/0 mon.b 2024-03-29T11:23:39.671 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-03-29T11:23:39.671 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-29T11:23:39.671 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.685+0000 7fb82bfff640 4 set_mon_vals no callback set 2024-03-29T11:23:39.672 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.687+0000 7fb83bc55640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:39.672 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.687+0000 7fb83bc55640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:39.672 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.687+0000 7fb83bc55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:23:39.672 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.687+0000 7fb83bc55640 1 -- 172.21.15.47:0/2952316097 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb81c005080 con 0x7fb8341457f0 2024-03-29T11:23:39.672 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.687+0000 7fb82a7fc640 1 -- 172.21.15.47:0/2952316097 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1261771619 0 0) 0x7fb82c003c90 con 0x7fb8341457f0 2024-03-29T11:23:39.672 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.688+0000 7fb82a7fc640 1 -- 172.21.15.47:0/2952316097 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3418925575 0 0) 0x7fb82c084cc0 con 0x7fb8341457f0 2024-03-29T11:23:39.672 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.688+0000 7fb82a7fc640 10 client.4664.objecter ms_dispatch 0x7fb834000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:23:39.672 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.688+0000 7fb82a7fc640 3 client.4664.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:23:39.672 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.688+0000 7fb82a7fc640 3 client.4664.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:23:39.672 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.688+0000 7fb8399f1640 10 monclient: get_auth_request con 0x7fb81806dd60 auth_method 0 2024-03-29T11:23:39.672 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.688+0000 7fb82a7fc640 20 client.4664.objecter dump_active .. 0 homeless 2024-03-29T11:23:39.672 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.688+0000 7fb82a7fc640 10 client.4664.objecter ms_handle_connect 0x7fb81806dd60 2024-03-29T11:23:39.675 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.695+0000 7fb82a7fc640 1 -- 172.21.15.47:0/2952316097 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb82c0503d0 con 0x7fb8341457f0 2024-03-29T11:23:39.675 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.695+0000 7fb82a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:39.675 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:39.695+0000 7fb82a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:40.007 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:40.027+0000 7fb83bc55640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:40.007 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:40.027+0000 7fb83bc55640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:40.007 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:40.027+0000 7fb83bc55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-03-29T11:23:40.007 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:40.027+0000 7fb83bc55640 1 -- 172.21.15.47:0/2952316097 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb81c005330 con 0x7fb8341457f0 2024-03-29T11:23:40.008 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:40.028+0000 7fb82a7fc640 1 -- 172.21.15.47:0/2952316097 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1178522771) 0x7fb82c0524d0 con 0x7fb8341457f0 2024-03-29T11:23:40.008 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:40.029+0000 7fb82a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:40.008 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:40.029+0000 7fb82a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:40.008 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-29T11:23:40.009 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":11,"fsid":"bb938893-5bcf-496d-8a49-24fbf9065510","created":"2024-03-29T11:12:40.995979+0000","modified":"2024-03-29T11:22:45.925385+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:22:45.925385+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bcf217-b930-4b2c-84c1-56b8b18c9805","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"bdab7243-0401-4284-ac2d-2802e712c07b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5594d0a6-3ad9-46ed-a21d-b9f4968b6d8c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"1ffc3ac2-c0ea-4099-b4db-eaeeb9f099a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1de122b-37a5-469f-bff2-bd60d934e4d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4ac1814f-2e59-4bce-a68b-83e395b1f724","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c06ac13d-25ef-476f-8acf-0f90ad4b2a00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"52478c03-0b80-454b-ba89-4a8228777210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:40.013 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:40.033+0000 7fb83bc55640 10 librados: watch_flush enter 2024-03-29T11:23:40.013 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:40.033+0000 7fb83bc55640 10 librados: watch_flush exit 2024-03-29T11:23:40.013 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:40.033+0000 7fb83bc55640 20 client.4664.objecter shutdown clearing up homeless session... 2024-03-29T11:23:40.013 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:40.033+0000 7fb83bc55640 10 client.4664.objecter successfully canceled tick 2024-03-29T11:23:40.013 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:40.033+0000 7fb83bc55640 1 -- 172.21.15.47:0/2952316097 >> v1:172.21.15.121:6816/1964117370 conn(0x7fb81806dd60 legacy=0x7fb818070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:40.013 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:40.033+0000 7fb83bc55640 10 monclient: shutdown 2024-03-29T11:23:40.013 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:40.033+0000 7fb83bc55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:40.013 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:40.033+0000 7fb83bc55640 1 -- 172.21.15.47:0/2952316097 >> v1:172.21.15.47:6790/0 conn(0x7fb8341457f0 legacy=0x7fb8341dab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:40.014 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:40.033+0000 7fb83bc55640 1 -- 172.21.15.47:0/2952316097 shutdown_connections 2024-03-29T11:23:40.014 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:40.033+0000 7fb83bc55640 1 -- 172.21.15.47:0/2952316097 >> 172.21.15.47:0/2952316097 conn(0x7fb834066a00 msgr2=0x7fb8340b2520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:40.014 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:40.033+0000 7fb83bc55640 1 -- 172.21.15.47:0/2952316097 shutdown_connections 2024-03-29T11:23:40.014 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:40.033+0000 7fb83bc55640 1 -- 172.21.15.47:0/2952316097 wait complete. 2024-03-29T11:23:40.014 INFO:teuthology.misc.health.smithi047.stderr:2024-03-29T11:23:40.033+0000 7fb83bc55640 1 librados: shutdown 2024-03-29T11:23:40.028 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:40.028 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:23:40.030 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-29T11:23:40.030 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-29T11:23:40.030 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:23:40.061 INFO:tasks.ceph.osd.0:Stopped 2024-03-29T11:23:40.061 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-29T11:23:40.061 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:23:40.139 INFO:tasks.ceph.osd.1:Stopped 2024-03-29T11:23:40.139 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-29T11:23:40.139 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:23:40.216 INFO:tasks.ceph.osd.2:Stopped 2024-03-29T11:23:40.287 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-29T11:23:40.288 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:23:40.323 INFO:tasks.ceph.osd.3:Stopped 2024-03-29T11:23:40.323 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-29T11:23:40.323 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:23:40.359 INFO:tasks.ceph.osd.4:Stopped 2024-03-29T11:23:40.360 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-29T11:23:40.360 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:23:40.429 INFO:tasks.ceph.osd.5:Stopped 2024-03-29T11:23:40.430 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-29T11:23:40.430 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:23:40.495 INFO:tasks.ceph.osd.6:Stopped 2024-03-29T11:23:40.495 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-29T11:23:40.495 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:23:40.574 INFO:tasks.ceph.osd.7:Stopped 2024-03-29T11:23:40.574 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-29T11:23:40.574 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-29T11:23:40.574 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:23:40.597 INFO:tasks.ceph.mgr.y:Stopped 2024-03-29T11:23:40.598 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-29T11:23:40.598 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:23:40.639 INFO:tasks.ceph.mgr.x:Stopped 2024-03-29T11:23:40.640 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-29T11:23:40.640 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-29T11:23:40.640 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:23:40.650 INFO:tasks.ceph.mon.a:Stopped 2024-03-29T11:23:40.651 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-29T11:23:40.651 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:23:40.701 INFO:tasks.ceph.mon.c:Stopped 2024-03-29T11:23:40.702 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-29T11:23:40.702 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:23:40.713 INFO:tasks.ceph.mon.b:Stopped 2024-03-29T11:23:40.713 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-29T11:23:40.713 DEBUG:teuthology.orchestra.run.smithi047:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | head -n 1 2024-03-29T11:23:40.773 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi047.front.sepia.ceph.com 2024-03-29T11:23:40.773 DEBUG:teuthology.orchestra.run.smithi047:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-29T11:23:43.155 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi047.front.sepia.ceph.com 2024-03-29T11:23:43.156 DEBUG:teuthology.orchestra.run.smithi047:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-29T11:23:43.254 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi047.front.sepia.ceph.com 2024-03-29T11:23:43.254 DEBUG:teuthology.orchestra.run.smithi047:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-29T11:23:43.313 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi047.front.sepia.ceph.com 2024-03-29T11:23:43.313 DEBUG:teuthology.orchestra.run.smithi047:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-29T11:23:43.362 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi121.front.sepia.ceph.com 2024-03-29T11:23:43.362 DEBUG:teuthology.orchestra.run.smithi121:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-29T11:23:46.405 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi121.front.sepia.ceph.com 2024-03-29T11:23:46.406 DEBUG:teuthology.orchestra.run.smithi121:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-29T11:23:46.490 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi121.front.sepia.ceph.com 2024-03-29T11:23:46.491 DEBUG:teuthology.orchestra.run.smithi121:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-29T11:23:46.558 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi121.front.sepia.ceph.com 2024-03-29T11:23:46.558 DEBUG:teuthology.orchestra.run.smithi121:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-29T11:23:46.617 INFO:tasks.ceph:Archiving mon data... 2024-03-29T11:23:46.619 DEBUG:teuthology.misc:Transferring archived files from smithi047:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622679/data/mon.a.tgz 2024-03-29T11:23:46.619 DEBUG:teuthology.orchestra.run.smithi047:> mktemp 2024-03-29T11:23:46.637 INFO:teuthology.orchestra.run.smithi047.stdout:/tmp/tmp.NzEdUGqKgR 2024-03-29T11:23:46.637 DEBUG:teuthology.orchestra.run.smithi047:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.NzEdUGqKgR 2024-03-29T11:23:46.740 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 0666 /tmp/tmp.NzEdUGqKgR 2024-03-29T11:23:46.829 DEBUG:teuthology.orchestra.remote:smithi047:/tmp/tmp.NzEdUGqKgR is 126KB 2024-03-29T11:23:46.881 DEBUG:teuthology.orchestra.run.smithi047:> rm -fr /tmp/tmp.NzEdUGqKgR 2024-03-29T11:23:46.895 DEBUG:teuthology.misc:Transferring archived files from smithi047:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622679/data/mon.c.tgz 2024-03-29T11:23:46.896 DEBUG:teuthology.orchestra.run.smithi047:> mktemp 2024-03-29T11:23:46.950 INFO:teuthology.orchestra.run.smithi047.stdout:/tmp/tmp.uJysq29gBX 2024-03-29T11:23:46.951 DEBUG:teuthology.orchestra.run.smithi047:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.uJysq29gBX 2024-03-29T11:23:47.039 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 0666 /tmp/tmp.uJysq29gBX 2024-03-29T11:23:47.123 DEBUG:teuthology.orchestra.remote:smithi047:/tmp/tmp.uJysq29gBX is 126KB 2024-03-29T11:23:47.173 DEBUG:teuthology.orchestra.run.smithi047:> rm -fr /tmp/tmp.uJysq29gBX 2024-03-29T11:23:47.187 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622679/data/mon.b.tgz 2024-03-29T11:23:47.187 DEBUG:teuthology.orchestra.run.smithi121:> mktemp 2024-03-29T11:23:47.205 INFO:teuthology.orchestra.run.smithi121.stdout:/tmp/tmp.cHPpLBCgI6 2024-03-29T11:23:47.205 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.cHPpLBCgI6 2024-03-29T11:23:47.311 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0666 /tmp/tmp.cHPpLBCgI6 2024-03-29T11:23:47.399 DEBUG:teuthology.orchestra.remote:smithi121:/tmp/tmp.cHPpLBCgI6 is 126KB 2024-03-29T11:23:47.422 DEBUG:teuthology.orchestra.run.smithi121:> rm -fr /tmp/tmp.cHPpLBCgI6 2024-03-29T11:23:47.437 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-29T11:23:47.437 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:23:47.440 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:23:47.519 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:23:47.559 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:23:47.560 INFO:tasks.ceph:Archiving crash dumps... 2024-03-29T11:23:47.561 DEBUG:teuthology.misc:Transferring archived files from smithi047:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622679/remote/smithi047/crash 2024-03-29T11:23:47.562 DEBUG:teuthology.orchestra.run.smithi047:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:23:47.589 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622679/remote/smithi121/crash 2024-03-29T11:23:47.590 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:23:47.626 INFO:tasks.ceph:Compressing logs... 2024-03-29T11:23:47.626 DEBUG:teuthology.orchestra.run.smithi047:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:23:47.634 DEBUG:teuthology.orchestra.run.smithi121:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:23:47.658 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-29T11:23:47.658 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41342.log 2024-03-29T11:23:47.658 INFO:teuthology.orchestra.run.smithi047.stderr:gzip/var/log/ceph/ceph-mon.a.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.39348.log 2024-03-29T11:23:47.698 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.41342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39153.log 2024-03-29T11:23:47.698 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.39348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41732.log 2024-03-29T11:23:47.698 INFO:teuthology.orchestra.run.smithi047.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41342.log.gz 2024-03-29T11:23:47.698 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.39153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40011.log 2024-03-29T11:23:47.698 INFO:teuthology.orchestra.run.smithi047.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39348.log.gz 2024-03-29T11:23:47.698 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.41732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39660.log 2024-03-29T11:23:47.698 INFO:teuthology.orchestra.run.smithi047.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39153.log.gz 2024-03-29T11:23:47.698 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.40011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40245.log 2024-03-29T11:23:47.698 INFO:teuthology.orchestra.run.smithi047.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41732.log.gz 2024-03-29T11:23:47.698 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.39660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40913.log 2024-03-29T11:23:47.698 INFO:teuthology.orchestra.run.smithi047.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40011.log.gz 2024-03-29T11:23:47.699 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.40245.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39660.log.gz 2024-03-29T11:23:47.699 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40362.log 2024-03-29T11:23:47.699 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.40913.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40245.log.gz 2024-03-29T11:23:47.699 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39504.log 2024-03-29T11:23:47.699 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.40362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38410.log 2024-03-29T11:23:47.699 INFO:teuthology.orchestra.run.smithi047.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40913.log.gz 2024-03-29T11:23:47.699 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.39504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38332.log 2024-03-29T11:23:47.699 INFO:teuthology.orchestra.run.smithi047.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40362.log.gz 2024-03-29T11:23:47.699 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41225.log 2024-03-29T11:23:47.699 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.38410.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39504.log.gz 2024-03-29T11:23:47.699 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.38332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39231.log 2024-03-29T11:23:47.699 INFO:teuthology.orchestra.run.smithi047.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.38410.log.gz 2024-03-29T11:23:47.699 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.41225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41537.log 2024-03-29T11:23:47.699 INFO:teuthology.orchestra.run.smithi047.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38332.log.gz 2024-03-29T11:23:47.699 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41576.log 2024-03-29T11:23:47.700 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.39231.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41225.log.gz 2024-03-29T11:23:47.700 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.41537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40796.log 2024-03-29T11:23:47.700 INFO:teuthology.orchestra.run.smithi047.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39231.log.gz 2024-03-29T11:23:47.700 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.41576.log: gzip -5 86.1% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.41537.log.gz /var/log/ceph/ceph-client.admin.41654.log 2024-03-29T11:23:47.700 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-29T11:23:47.700 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38880.log 2024-03-29T11:23:47.700 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.40796.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41576.log.gz 2024-03-29T11:23:47.700 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.41654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40991.log 2024-03-29T11:23:47.700 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.38880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40874.log 2024-03-29T11:23:47.700 INFO:teuthology.orchestra.run.smithi047.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40796.log.gz 2024-03-29T11:23:47.700 INFO:teuthology.orchestra.run.smithi047.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41654.log.gz 2024-03-29T11:23:47.700 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.40991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38958.log 2024-03-29T11:23:47.700 INFO:teuthology.orchestra.run.smithi047.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38880.log.gz 2024-03-29T11:23:47.701 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.40874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35019.log 2024-03-29T11:23:47.701 INFO:teuthology.orchestra.run.smithi047.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40991.log.gz 2024-03-29T11:23:47.701 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.38958.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:23:47.701 INFO:teuthology.orchestra.run.smithi047.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.40874.log.gz 2024-03-29T11:23:47.701 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.35019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38488.log 2024-03-29T11:23:47.701 INFO:teuthology.orchestra.run.smithi047.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.38958.log.gz 2024-03-29T11:23:47.701 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39270.log 2024-03-29T11:23:47.701 INFO:teuthology.orchestra.run.smithi047.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35019.log.gz 2024-03-29T11:23:47.701 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.38488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40404.log 2024-03-29T11:23:47.701 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.39270.log: 85.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.38488.log.gz 2024-03-29T11:23:47.701 INFO:teuthology.orchestra.run.smithi047.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39621.log 2024-03-29T11:23:47.701 INFO:teuthology.orchestra.run.smithi047.stderr: 93.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:23:47.701 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.40404.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39270.log.gz 2024-03-29T11:23:47.702 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40952.log 2024-03-29T11:23:47.702 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.39621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38607.log 2024-03-29T11:23:47.702 INFO:teuthology.orchestra.run.smithi047.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40404.log.gz 2024-03-29T11:23:47.702 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.40952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41186.log 2024-03-29T11:23:47.702 INFO:teuthology.orchestra.run.smithi047.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39621.log.gz 2024-03-29T11:23:47.702 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.38607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38997.log 2024-03-29T11:23:47.702 INFO:teuthology.orchestra.run.smithi047.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40952.log.gz 2024-03-29T11:23:47.702 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.41186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35122.log 2024-03-29T11:23:47.702 INFO:teuthology.orchestra.run.smithi047.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.38607.log.gz 2024-03-29T11:23:47.702 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.38997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39114.log 2024-03-29T11:23:47.702 INFO:teuthology.orchestra.run.smithi047.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41186.log.gz 2024-03-29T11:23:47.702 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.35122.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38997.log.gz 2024-03-29T11:23:47.702 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41147.log 2024-03-29T11:23:47.702 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.39114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39972.log 2024-03-29T11:23:47.703 INFO:teuthology.orchestra.run.smithi047.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35122.log.gz 2024-03-29T11:23:47.703 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.41147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41420.log 2024-03-29T11:23:47.703 INFO:teuthology.orchestra.run.smithi047.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39114.log.gz 2024-03-29T11:23:47.703 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.39972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41030.log 2024-03-29T11:23:47.703 INFO:teuthology.orchestra.run.smithi047.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41147.log.gz 2024-03-29T11:23:47.703 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.41420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40167.log 2024-03-29T11:23:47.703 INFO:teuthology.orchestra.run.smithi047.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.39972.log.gz 2024-03-29T11:23:47.703 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.41030.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-29T11:23:47.703 INFO:teuthology.orchestra.run.smithi047.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41420.log.gz 2024-03-29T11:23:47.703 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.40167.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41030.log.gz 2024-03-29T11:23:47.703 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40482.log 2024-03-29T11:23:47.703 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39894.log 2024-03-29T11:23:47.703 INFO:teuthology.orchestra.run.smithi047.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.40167.log.gz 2024-03-29T11:23:47.703 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.40482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34734.log 2024-03-29T11:23:47.703 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.39894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40718.log 2024-03-29T11:23:47.704 INFO:teuthology.orchestra.run.smithi047.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40482.log.gz 2024-03-29T11:23:47.704 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.34734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41069.log 2024-03-29T11:23:47.704 INFO:teuthology.orchestra.run.smithi047.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39894.log.gz 2024-03-29T11:23:47.704 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.40718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41771.log 2024-03-29T11:23:47.704 INFO:teuthology.orchestra.run.smithi047.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.34734.log.gz 2024-03-29T11:23:47.704 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.41069.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40718.log.gz 2024-03-29T11:23:47.704 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40679.log 2024-03-29T11:23:47.704 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38685.log 2024-03-29T11:23:47.704 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.41771.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41069.log.gz 2024-03-29T11:23:47.704 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.40679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41264.log 2024-03-29T11:23:47.704 INFO:teuthology.orchestra.run.smithi047.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41771.log.gz 2024-03-29T11:23:47.704 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.38685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40089.log 2024-03-29T11:23:47.704 INFO:teuthology.orchestra.run.smithi047.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40679.log.gz 2024-03-29T11:23:47.704 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.41264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41381.log 2024-03-29T11:23:47.705 INFO:teuthology.orchestra.run.smithi047.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38685.log.gz 2024-03-29T11:23:47.705 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.40089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41693.log 2024-03-29T11:23:47.705 INFO:teuthology.orchestra.run.smithi047.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41264.log.gz 2024-03-29T11:23:47.705 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.41381.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-29T11:23:47.705 INFO:teuthology.orchestra.run.smithi047.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40089.log.gz 2024-03-29T11:23:47.705 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.41693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40128.log 2024-03-29T11:23:47.705 INFO:teuthology.orchestra.run.smithi047.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41381.log.gz 2024-03-29T11:23:47.705 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39933.log 2024-03-29T11:23:47.705 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-osd.3.log: /var/log/ceph/ceph-client.admin.40128.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41693.log.gz 2024-03-29T11:23:47.705 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38527.log 2024-03-29T11:23:47.705 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.39933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39036.log 2024-03-29T11:23:47.705 INFO:teuthology.orchestra.run.smithi047.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40128.log.gz 2024-03-29T11:23:47.705 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.38527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39192.log 2024-03-29T11:23:47.705 INFO:teuthology.orchestra.run.smithi047.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39933.log.gz 2024-03-29T11:23:47.706 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.39036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40561.log 2024-03-29T11:23:47.706 INFO:teuthology.orchestra.run.smithi047.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38527.log.gz 2024-03-29T11:23:47.706 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.39192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40323.log 2024-03-29T11:23:47.706 INFO:teuthology.orchestra.run.smithi047.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39036.log.gz 2024-03-29T11:23:47.706 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.40561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40284.log 2024-03-29T11:23:47.706 INFO:teuthology.orchestra.run.smithi047.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39192.log.gz 2024-03-29T11:23:47.706 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.40323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40206.log 2024-03-29T11:23:47.706 INFO:teuthology.orchestra.run.smithi047.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.40561.log.gz 2024-03-29T11:23:47.706 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.40284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39777.log 2024-03-29T11:23:47.706 INFO:teuthology.orchestra.run.smithi047.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40323.log.gz 2024-03-29T11:23:47.706 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.40206.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40284.log.gz 2024-03-29T11:23:47.706 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41108.log 2024-03-29T11:23:47.706 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38724.log 2024-03-29T11:23:47.706 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.39777.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40206.log.gz 2024-03-29T11:23:47.707 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.41108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38566.log 2024-03-29T11:23:47.707 INFO:teuthology.orchestra.run.smithi047.stderr: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.39777.log.gz 2024-03-29T11:23:47.707 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.38724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38802.log 2024-03-29T11:23:47.707 INFO:teuthology.orchestra.run.smithi047.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41108.log.gz 2024-03-29T11:23:47.707 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.38566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40443.log 2024-03-29T11:23:47.707 INFO:teuthology.orchestra.run.smithi047.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.38724.log.gz 2024-03-29T11:23:47.707 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.38802.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-29T11:23:47.707 INFO:teuthology.orchestra.run.smithi047.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38566.log.gz 2024-03-29T11:23:47.707 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.40443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40600.log 2024-03-29T11:23:47.707 INFO:teuthology.orchestra.run.smithi047.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.38802.log.gz 2024-03-29T11:23:47.707 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39387.log 2024-03-29T11:23:47.707 INFO:teuthology.orchestra.run.smithi047.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40443.log.gz 2024-03-29T11:23:47.707 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.40600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39543.log 2024-03-29T11:23:47.707 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.39387.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40600.log.gz 2024-03-29T11:23:47.708 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41498.log 2024-03-29T11:23:47.708 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.39543.log: gzip -5 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39387.log.gz 2024-03-29T11:23:47.708 INFO:teuthology.orchestra.run.smithi047.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39816.log 2024-03-29T11:23:47.708 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.41498.log: gzip 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39543.log.gz 2024-03-29T11:23:47.708 INFO:teuthology.orchestra.run.smithi047.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40757.log 2024-03-29T11:23:47.708 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.39816.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.41498.log.gz 2024-03-29T11:23:47.708 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39582.log 2024-03-29T11:23:47.708 INFO:teuthology.orchestra.run.smithi047.stderr: 85.6%/var/log/ceph/ceph-client.admin.40757.log: -- replaced with /var/log/ceph/ceph-client.admin.39816.log.gz 2024-03-29T11:23:47.708 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-29T11:23:47.708 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.39582.log: gzip -5 --verbose 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40757.log.gz 2024-03-29T11:23:47.708 INFO:teuthology.orchestra.run.smithi047.stderr: -- /var/log/ceph/ceph-client.admin.39465.log 2024-03-29T11:23:47.708 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-osd.0.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39582.log.gz 2024-03-29T11:23:47.708 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40835.log 2024-03-29T11:23:47.708 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.39465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39855.log 2024-03-29T11:23:47.709 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.40835.log: 85.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40521.log 2024-03-29T11:23:47.709 INFO:teuthology.orchestra.run.smithi047.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39465.log.gz 2024-03-29T11:23:47.709 INFO:teuthology.orchestra.run.smithi047.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40835.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41615.log 2024-03-29T11:23:47.709 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-29T11:23:47.709 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.39855.log: /var/log/ceph/ceph-client.admin.40521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41303.log 2024-03-29T11:23:47.709 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.41615.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.39855.log.gz 2024-03-29T11:23:47.709 INFO:teuthology.orchestra.run.smithi047.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40521.log.gz 2024-03-29T11:23:47.709 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38371.log 2024-03-29T11:23:47.709 INFO:teuthology.orchestra.run.smithi047.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41615.log.gz 2024-03-29T11:23:47.709 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:23:47.709 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.41303.log: /var/log/ceph/ceph-client.admin.38371.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41303.log.gz 2024-03-29T11:23:47.709 INFO:teuthology.orchestra.run.smithi047.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38371.log.gz 2024-03-29T11:23:47.709 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39426.log 2024-03-29T11:23:47.709 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38841.log 2024-03-29T11:23:47.710 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.39426.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-29T11:23:47.710 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.38841.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39426.log.gz 2024-03-29T11:23:47.710 INFO:teuthology.orchestra.run.smithi047.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.38841.log.gz 2024-03-29T11:23:47.710 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38646.log 2024-03-29T11:23:47.710 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38449.log 2024-03-29T11:23:47.710 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.38646.log: 92.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:23:47.710 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40640.log 2024-03-29T11:23:47.710 INFO:teuthology.orchestra.run.smithi047.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.38646.log.gz 2024-03-29T11:23:47.710 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.38449.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28909.log 2024-03-29T11:23:47.710 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.40640.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38449.log.gz 2024-03-29T11:23:47.710 INFO:teuthology.orchestra.run.smithi047.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-29T11:23:47.710 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40050.log 2024-03-29T11:23:47.710 INFO:teuthology.orchestra.run.smithi047.stderr: 85.3% -- replaced with /var/log/ceph/ceph-client.admin.40640.log.gz 2024-03-29T11:23:47.711 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38763.log 2024-03-29T11:23:47.711 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph.tmp-client.admin.28909.log: /var/log/ceph/ceph-client.admin.40050.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28909.log.gz 2024-03-29T11:23:47.711 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39309.log 2024-03-29T11:23:47.711 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.38763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38919.log 2024-03-29T11:23:47.711 INFO:teuthology.orchestra.run.smithi047.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40050.log.gz 2024-03-29T11:23:47.711 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.39309.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.38763.log.gz 2024-03-29T11:23:47.711 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41459.log 2024-03-29T11:23:47.711 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.38919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39738.log 2024-03-29T11:23:47.711 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.41459.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39309.log.gz 2024-03-29T11:23:47.711 INFO:teuthology.orchestra.run.smithi047.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.38919.log.gz 2024-03-29T11:23:47.711 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39699.log 2024-03-29T11:23:47.711 INFO:teuthology.orchestra.run.smithi047.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41459.log.gz 2024-03-29T11:23:47.711 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39075.log 2024-03-29T11:23:47.711 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.39738.log: /var/log/ceph/ceph-client.admin.39699.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-29T11:23:47.712 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.39075.log: 85.9% 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39738.log.gz 2024-03-29T11:23:47.712 INFO:teuthology.orchestra.run.smithi047.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39699.log.gz 2024-03-29T11:23:47.712 INFO:teuthology.orchestra.run.smithi047.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39075.log.gz 2024-03-29T11:23:47.719 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-29T11:23:47.719 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34467.log 2024-03-29T11:23:47.719 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34747.log 2024-03-29T11:23:47.719 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.34467.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:23:47.719 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.34747.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-29T11:23:47.719 INFO:teuthology.orchestra.run.smithi121.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34467.log.gz 2024-03-29T11:23:47.719 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34627.log 2024-03-29T11:23:47.719 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph.audit.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34747.log.gz 2024-03-29T11:23:47.719 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-29T11:23:47.719 INFO:teuthology.orchestra.run.smithi121.stderr: 88.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:23:47.719 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.34627.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-29T11:23:47.719 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34507.log 2024-03-29T11:23:47.720 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-osd.7.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34627.log.gz 2024-03-29T11:23:47.720 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-29T11:23:47.720 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.34507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34667.log 2024-03-29T11:23:47.720 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34547.log 2024-03-29T11:23:47.720 INFO:teuthology.orchestra.run.smithi121.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34507.log.gz 2024-03-29T11:23:47.720 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.34667.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:23:47.720 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-29T11:23:47.720 INFO:teuthology.orchestra.run.smithi121.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34667.log.gz/var/log/ceph/ceph.log: 2024-03-29T11:23:47.720 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.34547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34587.log 2024-03-29T11:23:47.720 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-mon.b.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34547.log.gz 2024-03-29T11:23:47.720 INFO:teuthology.orchestra.run.smithi121.stderr: 87.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:23:47.720 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34707.log 2024-03-29T11:23:47.720 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.34587.log: /var/log/ceph/ceph-client.admin.34707.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34587.log.gz 2024-03-29T11:23:47.720 INFO:teuthology.orchestra.run.smithi121.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34707.log.gz 2024-03-29T11:23:47.772 INFO:teuthology.orchestra.run.smithi121.stderr: 92.3% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-29T11:23:59.851 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-mon.c.log: 94.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-29T11:23:59.889 INFO:teuthology.orchestra.run.smithi047.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-29T11:23:59.913 INFO:teuthology.orchestra.run.smithi121.stderr: 94.8% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-29T11:24:00.030 INFO:teuthology.orchestra.run.smithi121.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-29T11:24:04.017 INFO:teuthology.orchestra.run.smithi121.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-29T11:24:04.128 INFO:teuthology.orchestra.run.smithi047.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-29T11:24:04.197 INFO:teuthology.orchestra.run.smithi047.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-29T11:24:04.275 INFO:teuthology.orchestra.run.smithi121.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-29T11:24:40.357 INFO:teuthology.orchestra.run.smithi047.stderr: 92.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-29T11:24:44.338 INFO:teuthology.orchestra.run.smithi047.stderr: 92.9% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-29T11:24:44.340 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-29T11:24:44.340 INFO:teuthology.orchestra.run.smithi047.stderr:real 0m56.692s 2024-03-29T11:24:44.340 INFO:teuthology.orchestra.run.smithi047.stderr:user 2m41.183s 2024-03-29T11:24:44.340 INFO:teuthology.orchestra.run.smithi047.stderr:sys 0m5.255s 2024-03-29T11:24:54.319 INFO:teuthology.orchestra.run.smithi121.stderr: 92.7% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-29T11:24:54.320 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-29T11:24:54.320 INFO:teuthology.orchestra.run.smithi121.stderr:real 1m6.632s 2024-03-29T11:24:54.320 INFO:teuthology.orchestra.run.smithi121.stderr:user 1m59.483s 2024-03-29T11:24:54.320 INFO:teuthology.orchestra.run.smithi121.stderr:sys 0m4.428s 2024-03-29T11:24:54.321 INFO:tasks.ceph:Archiving logs... 2024-03-29T11:24:54.321 DEBUG:teuthology.misc:Transferring archived files from smithi047:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622679/remote/smithi047/log 2024-03-29T11:24:54.325 DEBUG:teuthology.orchestra.run.smithi047:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:25:17.627 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622679/remote/smithi121/log 2024-03-29T11:25:17.628 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:25:36.023 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:25:36.228 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=8e516f4e1a7a4ab1b783d1ec6e0dabb4 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:25:36.232 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-29T11:25:36.246 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-29T11:25:36.261 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-29T11:25:36.261 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:25:36.275 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:25:36.350 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-29T11:25:36.359 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-29T11:25:36.359 DEBUG:teuthology.orchestra.run.smithi047:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:25:36.362 DEBUG:teuthology.orchestra.run.smithi121:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:25:36.375 INFO:teuthology.orchestra.run.smithi047.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:25:36.378 INFO:teuthology.orchestra.run.smithi121.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:25:36.380 INFO:teuthology.orchestra.run.smithi047.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:25:36.380 INFO:teuthology.orchestra.run.smithi047.stdout:=============================================================================== 2024-03-29T11:25:36.381 INFO:teuthology.orchestra.run.smithi047.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 55 +421us[ +426us] +/- 85ms 2024-03-29T11:25:36.381 INFO:teuthology.orchestra.run.smithi047.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 55 -142us[ -138us] +/- 80ms 2024-03-29T11:25:36.381 INFO:teuthology.orchestra.run.smithi047.stdout:^* hv03.front.sepia.ceph.com 3 6 377 54 -247us[ -243us] +/- 79ms 2024-03-29T11:25:36.381 INFO:teuthology.orchestra.run.smithi047.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:25:36.382 INFO:teuthology.orchestra.run.smithi121.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:25:36.382 INFO:teuthology.orchestra.run.smithi121.stdout:=============================================================================== 2024-03-29T11:25:36.383 INFO:teuthology.orchestra.run.smithi121.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 53 +437us[ +437us] +/- 85ms 2024-03-29T11:25:36.383 INFO:teuthology.orchestra.run.smithi121.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 56 -129us[ -129us] +/- 80ms 2024-03-29T11:25:36.383 INFO:teuthology.orchestra.run.smithi121.stdout:^* hv03.front.sepia.ceph.com 3 6 377 52 -243us[ -244us] +/- 80ms 2024-03-29T11:25:36.383 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:25:36.384 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-29T11:25:36.397 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-29T11:25:36.398 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-29T11:25:36.412 DEBUG:teuthology.orchestra.run.smithi047:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:25:36.492 DEBUG:teuthology.orchestra.run.smithi121:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:25:36.561 DEBUG:teuthology.orchestra.run.smithi047:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:25:36.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:25:36.592 DEBUG:teuthology.orchestra.run.smithi121:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:25:36.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:25:36.623 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-29T11:25:36.632 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-29T11:25:36.642 INFO:teuthology.task.internal:Duration was 1220.812648 seconds 2024-03-29T11:25:36.642 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-29T11:25:36.652 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-29T11:25:36.652 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:25:36.655 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:25:36.697 INFO:teuthology.orchestra.run.smithi047.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:25:36.707 INFO:teuthology.orchestra.run.smithi121.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:25:37.079 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-29T11:25:37.079 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi047.front.sepia.ceph.com 2024-03-29T11:25:37.080 DEBUG:teuthology.orchestra.run.smithi047:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:25:37.101 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi121.front.sepia.ceph.com 2024-03-29T11:25:37.101 DEBUG:teuthology.orchestra.run.smithi121:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:25:37.129 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-29T11:25:37.129 DEBUG:teuthology.orchestra.run.smithi047:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:25:37.144 DEBUG:teuthology.orchestra.run.smithi121:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:25:37.197 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-29T11:25:37.197 DEBUG:teuthology.orchestra.run.smithi047:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:25:37.216 DEBUG:teuthology.orchestra.run.smithi121:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:25:37.333 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-29T11:25:37.343 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-29T11:25:37.343 DEBUG:teuthology.orchestra.run.smithi047:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:25:37.371 DEBUG:teuthology.orchestra.run.smithi121:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:25:37.398 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-29T11:25:37.407 DEBUG:teuthology.orchestra.run.smithi047:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:25:37.414 DEBUG:teuthology.orchestra.run.smithi121:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:25:37.437 INFO:teuthology.orchestra.run.smithi047.stdout:kernel.core_pattern = core 2024-03-29T11:25:37.463 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern = core 2024-03-29T11:25:37.492 DEBUG:teuthology.orchestra.run.smithi047:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:25:37.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:25:37.512 DEBUG:teuthology.orchestra.run.smithi121:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:25:37.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:25:37.546 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-29T11:25:37.556 INFO:teuthology.task.internal:Transferring archived files... 2024-03-29T11:25:37.557 DEBUG:teuthology.misc:Transferring archived files from smithi047:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622679/remote/smithi047 2024-03-29T11:25:37.558 DEBUG:teuthology.orchestra.run.smithi047:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:25:37.593 DEBUG:teuthology.misc:Transferring archived files from smithi121:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622679/remote/smithi121 2024-03-29T11:25:37.594 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:25:37.628 INFO:teuthology.task.internal:Removing archive directory... 2024-03-29T11:25:37.628 DEBUG:teuthology.orchestra.run.smithi047:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:25:37.630 DEBUG:teuthology.orchestra.run.smithi121:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:25:37.678 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-29T11:25:37.691 INFO:teuthology.task.internal:Not uploading archives. 2024-03-29T11:25:37.692 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-29T11:25:37.703 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-29T11:25:37.704 DEBUG:teuthology.orchestra.run.smithi047:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:25:37.706 DEBUG:teuthology.orchestra.run.smithi121:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:25:37.718 INFO:teuthology.orchestra.run.smithi047.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:25 /home/ubuntu/cephtest 2024-03-29T11:25:37.733 INFO:teuthology.orchestra.run.smithi121.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:25 /home/ubuntu/cephtest 2024-03-29T11:25:37.734 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-29T11:25:37.744 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-29T11:25:37.775 INFO:teuthology.nuke:Checking targets against current locks 2024-03-29T11:25:37.804 DEBUG:teuthology.nuke:shortname: smithi047 2024-03-29T11:25:37.804 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:25:37.825 DEBUG:teuthology.nuke:shortname: smithi121 2024-03-29T11:25:37.825 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:25:37.830 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi047.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622679', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:56:31.412883', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPNhVwhJk+rZpOqy4vRT+1shz5VX6cJK93bgh6UFH/YgmVvQNabahQprKdUnKhgDVpyMN5spp1rihixflu+VieQ='} 2024-03-29T11:25:37.847 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi121.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622679', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:56:31.410886', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMS5EAtFB8SYN2sAGTqa/rrYY+xvMfYBmHUjp0wioVu/18xT7z+V0ACprXjanFGxqrTGH9apissqBCVGhTCTp84='} 2024-03-29T11:25:37.876 INFO:teuthology.orchestra.console.smithi047:Power off 2024-03-29T11:25:37.877 DEBUG:teuthology.orchestra.console.smithi047:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:25:37.903 DEBUG:teuthology.orchestra.console.smithi047:power off output: Chassis Power Control: Down/Off 2024-03-29T11:25:37.903 DEBUG:teuthology.orchestra.console.smithi047:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:25:37.917 INFO:teuthology.orchestra.console.smithi121:Power off 2024-03-29T11:25:37.917 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:25:37.927 DEBUG:teuthology.orchestra.console.smithi047:check power output: Chassis Power is on 2024-03-29T11:25:37.940 DEBUG:teuthology.orchestra.console.smithi121:power off output: Chassis Power Control: Down/Off 2024-03-29T11:25:37.941 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:25:37.964 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is on 2024-03-29T11:25:41.928 DEBUG:teuthology.orchestra.console.smithi047:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:25:41.965 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:25:42.038 DEBUG:teuthology.orchestra.console.smithi047:check power output: Chassis Power is on 2024-03-29T11:25:42.074 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is on 2024-03-29T11:25:46.042 DEBUG:teuthology.orchestra.console.smithi047:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:25:46.076 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:25:46.153 DEBUG:teuthology.orchestra.console.smithi047:check power output: Chassis Power is off 2024-03-29T11:25:46.188 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is on 2024-03-29T11:25:46.254 INFO:teuthology.orchestra.console.smithi047:Power off completed 2024-03-29T11:25:50.190 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:25:50.301 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is off 2024-03-29T11:25:50.401 INFO:teuthology.orchestra.console.smithi121:Power off completed 2024-03-29T11:25:50.506 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} duration: 1220.8126482963562 failure_reason: reached maximum tries (91) after waiting for 540 seconds flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=8e516f4e1a7a4ab1b783d1ec6e0dabb4 status: fail success: false 2024-03-29T11:25:50.506 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:25:50.584 INFO:teuthology.run:FAIL